AD-A225  666 


r 


i 


ARI  Research  Note  90-44 

0110  hi!  copy 

Protocol  Analysis  of  Expert/Novice 
Command  Decision  Making 
During  Simulated  Fire 
Ground  Incidents 

Roberta  Calderwood,  Beth  W.  Crandall, 
and  Timothy  H.  Baynes 

Klein  Associates,  Inc. 


for 


Contracting  Officer’s  Representative 
Judith  Orasanu 


Basic  Research 
Michael  Kaplan,  Director 


July  1990 


V  '  ;  •  71  y”H 


i.  t. 


United  States  Army 

Research  Institute  for  the  Behavioral  and  Social  Sciences 


Approved  for  public  release;  distribution  Is  unlimited 

</0  Uti  :l',‘  129 


U.S.  ARMY  RESEARCH  INSTITUTE 

FOR  THE  BEHAVIORAL  AND  SOCIAL  SCIENCES 


A  Field  Operating  Agency  Under  the  Jurisdiction 
of  the  Deputy  Chief  of  Staff  for  Personnel 


EDGAR  M.  JOHNSON 
Technical  Director 


JON  W.  BLADES 
COL,  IN 
Commanding 


Research  accomplished  under  contract  for 
the  Department  of  the  Army 

Klein  Associates,  Inc. 

Technical  review  by 

George  Lawton 


NOTICES 


DISTRIBUTION:  This  report  has  been  cleared  for  release  to  the  Defense  Technical  Information 
Center  (DTIC)  to  comply  with  regulatory  requirements.  It  has  been  given  no  primary  distribution 
other  than  to  DTIC  and  will  be  available  only  through  DTIC  or  the  National  Technical 
Information  Service  (NT1S). 

FINAL  DISPOSITION:  This  report  may  be  destroyed  when  it  is  no  longer  needed.  Please  do  not 
return  it  to  (he  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences. 

NOTE:  The  views,  opinions,  and  findings  in  this  report  are  those  of  the  authors)  and  should  not 
be  construed  as  an  official  Department  of  the  Army  position,  policy,  or  decision,  unless  so 
designated  by  other  authorized  documents. 


UNCLASSIFIED 


REPORT  DOCUMENTATION  PAGE 


la.  REPORT  SECURITY  CLASSIFICATION 
Unclassified 


2a.  SECURITY  CLASSIFICATION  AUTHORITY 


2b.  DECLASSIFICATION  /DOWNGRADING  SCHEOULE 


Form  Approved 
OMB  NO.  0704  0188 


1b  RESTRICTIVE  MARKINGS 


3.  DISTRIBUTION /AVAILABILITY  OF  REPORT 
Approved  for  public  release; 
distribution  is  unlimited. 


4.  PERFORMING  ORGANIZATION  REPORT  NUMBER(S) 


6a.  NAME  OF  PERFORMING  ORGANIZATION 
Klein  Associates,  Inc. 


6c  ADDRESS  {City,  Stttt,  end  ZIP  Code) 
800  Livermore  Street 
Yellow  Springs,  OH  45387 


6b.  OFFICE  SYMBOL 
(if  ippliable) 


5.  MONITORING  ORGANIZATION  REPORT  NUMBER(S) 
ARI  Research  Note  90-44 


7a.  NAME  OF  MONITORING  ORGANIZATION 
U.S.  Army  Research  Institute 
Office  of  Basic  Research 


7b.  ADDRESS  (City.  Stitt,  ind  ZIP  Codt) 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333-5600 


8a.  NAME  OF  FUNDING  ^SPONSORING  .  8b  OFFICE  SYM80L 

organization  U.S.  Army  Research  (if  ippiicable) 
Institute  for  the  Behavioral 
and  Social  Sciences  PERI-BR 


8c  ADDRESS  (City,  Stitt,  ind  HP  Code) 
Office  of  Basic  Research 

5001  Eisenhower  Avenue 
Alexandria,  VA  22333-5600 


1 1 .  TITLE  (Include  Stcurity  Clisiification)  ,  ,  , 

Protocol  Analysis  of  Expert/Novice  Command  Decision  Making  During  Simulated  Fire  Ground 

Incidents 


9.  PROCUREMENT  INSTRUMENT  IDENTIFICATION  NUMBER 

KDA903-85-C-0327 


10.  SOURCE  OF  FUNDING  NUMBERS 


PROGRAM 
ELEMENT  NO. 

61102B 


PROJECT 

Task 

NO 

NO. 

74F 

WORK  UNIT 
ACCESSION  NO 


12.  PERSONAL  AUTHOR(S)  ,  T  , 

Calderwood,  Roberta;  Crandall,  Beth  W.;  and  Baynes,  Timothy  H.  (Klein  Associates,  Inc.) 


13a.  TYPE  OF  REPORT 
Interim 


16.  SUPPLEMENTARY  NOTATION 


13b  TIME  COVERED 
from  95/07  to  88/07 


14  DATE  OF  REPORT  (Yetr.  Month,  Oty)  IS.  PAGE  COUNT 

1990,  July  66 


COSATl  codes 


GROUP  SUB-GROUP 


H 


18.  SUBJECT  TERMS  (Continue  on  reverse  if  necessary  end  identify  by  block  number) 
Decision  making 
Reasoning,  /-- 
ExpertTse  ,  • '  C-  ,(  — — - 


19.  A8STRACT  ( Continue  on  reverse  if  necetfdry  end  identify  by  block  number) 

This  research  is  part  of  a  series  of  studies  that  investigate  how  decisions  are  made  in 

operational  settings  by  trained  personnel.  The  focus  is  on  environments  in  which  strategic 
and  tactical  decisions  must  be  made  under  conditions  of  uncertainty,  risk,  and  time  pressure, 
such  as  urban  firefighting.  Verbal  protocols  were  obtained  from  professional  urban  fire¬ 
fighters  during  simulated  incidents  in  which  they  were  asked  to  assume  the  role  of  fire 
ground  commander.  This  approach  afforded  an  opportunity  to  examine  decision  making  in  an 
ecologically  valid  way  and  to  obtain  a  rich  source  of  data  for  addressing  issues  of  infer¬ 
ence,  expectancies,  and  tacit  knowledge  that  are  part  of  decision  making  in  real-world 
settings. 


20.  DISTRIBUTION /AVAILABILITY  OF  ABSTRACT 
□  UNCLASSIFIED/UNLIMITED  □  SAME  AS  RPT 


22*.  NAME  O?  RESPONSIBLE  INDIVIDUAL 
Judith  Orasanu 


00  Form  1473,  JUN  86 


21  ABSTRACT  SECURITY  CLASSIFICATION 
□  OTIC  USERS  Unclassified 


22b  TELEPHONE  (Include  ArtiCooe)  22c  OFFICE  SYMBOL 

(202)  274-8722  PERI-BR 


Previous  editions  ire  obsolete. 


SECURITY  CLASSIFICATION  OF  THIS  PAGE 
UNCLASSIFIED 


ACKNOWLEDGMENTS 


This  study  has  been  supported  by  the  U.S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences.  The  authors  wish  to  thank  Dr.  Judith  Orasanu 
for  her  encouragement  and  support.  Our  deepest  appreciation  goes  to  the  per¬ 
sonnel  of  the  Dayton  Fire  Department  who  participated  in  the  study.  Thanks 
also  to  Jim  Nutt,  Chief  of  Miami  Tovnshlp  Fire  Department,  for  his  help  in 
developing  the  simulations. 


ill 


PROTOCOL  ANALYSIS  OP  BXPERT/NOVICB  COMMAND  DECISION  MAKING  DURING  SIMULATED 
FIRE  GROUND  INCIDENTS 


INTENT 


Page 

INTRODUCTION  .  1-1 

RECOGNITION-PRIMED  DECISION  MAKING  .  1-2 

Serial  Evaluation  .  1-4 

Progressive  Deepening  .  1-6 

Situation  Assessment .  1-7 

Study  Goals .  1-10 

Development  of  Urban  Fire  Ground  Command  Simulations 

(FGC-Simulation)  .  1-11 

METHOD .  1-14 

Study  Participants .  1-14 

Procedure .  1-15 

PROTOCOL  ANALYSIS .  1-17 

Coding  Procedures  .  1-17 

Coding  Reliability . .  1-20 

RESULTS .  1-25 

Analysis  of  Category  Frequencies .  1-25 

Content-Node  Analysis  of  Remark  Categories .  1-33 

Probe  Point  1.1 .  1-36 

Probe  Point  2.1 .  1-41 

Evidence  for  Decision  Strategies .  1-45 

SUMMARY  AND  CONCLUSIONS .  1-49 

REFERENCES .  1-53 

APPENDIX  A.  PROBE  POINTS.  .  A-l 


v 


LIST  OP  TABLES 


Table  1.  Sample- -coded  protocol 


2.  Percent  agreement  for  remark,  segments  for 
coders  A ,  B ,  and  C  .  .  .  . 


3.  Reliability  of  remark  category  coding: 
Kappa  coefficients  . 


4.  Average  number  of  remarks  for  experts  and 
novices  for  three  probe  points  . 


5.  Average  percent  of  decision  remark  categories 
for  experts  and  novices  for  three  probe  points 
(N-22)  . 


6.  Instances  of  situation  assessment  deliberation 


7.  Instances  of  action  deliberations. 


LIST  OF  FIGURES 


Figure  1.  Recognition-Primed  Decision  (RPD)  Model 


2.  Percent  of  cue-deliberation  and  action-deliberation 
for  experts  and  novices  in  selected  probe  points.  . 


3.  Cue  and  knovledge  probe  point  1.1 


4.  Action  and  goal  nodes  for  probe  point  1.1 


5.  Cue  and  knovledge  nodes  for  probe  point  2.1 


6.  Action  and  goal  nodes  for  probe  point  2.1 


i 


PROTOCOL  ANALYSIS  OF  EXPERT/NOVICE  COMMAND  DECISION 
MAKING  DURING  SIMULATED  FIRE  GROUND  INCIDENTS 


INTRODUCTION 

This  research  is  part  of  a  series  of  studies  investigating  how  decisions 
ore  made  in  operational  settings  by  trained  personnel.  Our  focus  has  been  ori 
environments  in  which  strategic  and  tactical  decisions  must  be  made  under 
conditions  of  uncertainty,  risk,  and  time  pressure,  such  as  urban  firefighting 
(Calderwood,  Crandall,  &  Klein,  1987;  Klein,  Calderwood.  &.  Clinton-Cirocco, 
198G),  wildland  firefighting  (Taynor,  Klein,  &  Thordsen,  1987),  and  tank 
platoon  battle  management  (Brezovic,  Klein,  4.  Thordsen,  1987). 

J'or  the  present  study,  verbal  protocols  were  obtained  from  professional 
urban  firefighters  during  simulated  fire  incidents  in  which  they  were  asked  to 
assume  the  role  of  fire  ground  commander  (FTjC).  The  PGC  is  responsible  for 
establishing  strategy  and  overseeing  tactical  maneuvering  of  personnel  and 
equipment  in  response  to  a  fire  emergency.  Decisions  include  where  and  how  to 
attack  the  fire  given  such  factors  as  risk  to  crews  and  civilians, 
availability  of  water  and  other  resources,  and  risk  to  exposed  property.  A 
major  goal  of  the  simulation  development  was  that  scenarios  be  realistic  and 
complex  enough  to  ensure  a  high  level  of  engagement.  Thus,  the  approach 
afforded  an  opportunity  to  examine  decision  making  in  a  more  ecologically 
valid  way  than  has  generally  been  the  case  in  decision  research  (Neisser, 

1976) ,  and  to  obtain  a  rich  and  distinctive  source  of  data  for  addressing 
issues  of  infeience,  expectancies,  and  tacit  knowledge  that  are  part  of 
decision  making  in  real-world  settings. 
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Although  the  participants  were  all  experienced  firefighters  who  had  from 
7  to  27  years  with  the  department,  we  were  particularly  interested  in  how 
different  levels  of  experience  and  skill  might  influence  the  nature  of  the 
decision  processes  that  would  be  reported.  Therefore,  officers  were  selected 
who  represented  a  wide  range  of  experience  and  ability. 

Surprisingly  little  attention  has  been  given  to  how  expertise  in  decision 
making  develops.  In  fact,  since  the  landmark  articles  by  Daniel  Kahneman  and 
Amos  Tversky  in  the  1970’s  (Kahneman  A  Tversky,  1972,  1973;  Tversky  4 
Kahneman,  1971,  1973,  1974),  decision  researchers  have  tended  to  stress  almost 
exclusively  the  ways  in  which  decision  makers  are  biased  and  suboptimal  (see 
Christensen-Szalanski  A  Beach,  1984;  Hammond,  1987;  Lopes,  1987).  We  think 
this  tendency  has  been  misguided  and  that  skill  in  decision  making  develops, 
as  it  does  in  other  human  endeavors,  in  terms  of  the  representation  of  domain- 
specific  knowledge  (e.g.,  Dreyfus  A  Dreyfus,  1986;  Glaser,  1981). 

The  method  of  protocol  analysis  developed  for  this  study,  along  with  the 
specific  study  questions  addressed  can  best  be  understood  in  relation  to  the 
descriptive  decision  model  that  we  are  developing  within  this  program  of 
research.  Therefore,  this  model  will  be  briefly  described  and  some  of  the 
findings  and  limitations  of  previous  studies  will  then  be  reviewed. 

Recognition-Primed  Decision  Making 
The  previously  cited  studies  of  command-and-control  decision  making 
relied  on  a  variant  of  Flanagan’s  (1954)  critical  incident  method  that  we  have 
called  the  Critical  Decision  method  (see  Klein,  Calderwood,  A  MacGregor,  in 
press).  The  method  uses  a  timeline  reconstruction  of  a  specific  incident  and 
focused  probes  to  obtain  retrospective  protocols  describing  a  decision  event. 
Questions  relate  to  the  cues  and  knowledge  that  were  heeded,  the  goals  that 
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were  operating,  any  alternatives  that  were  generated,  and  how  the  implemented 
course  of  action  was  selected.  Although  there  are  obvious  limitations  to  the 
method  related  to  people’s  ability  to  recall  and  verbalize  their  own  reasoning 
processes  (e.g.,  Nisbett  &  Wilson,  1977),  it  is  well  recognized  that  such 
protocols  can  provide  valuable  insights  about  consciousness  that  would 
otherwise  be  impossible  to  obtain  (Ericsson  &  Simon,  1984). 

In  these  studies  we  have  interviewed  and  observed  over  100  individuals 
and  probed  almost  400  decision  points.  This  extensive  data  base  has  provided 
a  rich  source  of  information  about  areas  in  which  current  models  of  decision 
making  may  be  inadequate  or  misleading  and  has  pointed  toward  potentially 
fruitful  avenues  for  further  research. 

One  of  the  most  striking  and  consistent  findings  across  these  studies  is 
how  little  evidence  was  found  for  evaluation  strategies  that  rely  on  a  direct 
comparison  of  the  strengths  and  weaknesses  of  a  set  of  generated  options. 
Instead,  experienced  decision  makers  are  most  frequently  found  to  rely  on 
their  abilities  to  quickly  classify  a  situation  on  the  basis  of  their  prior 
experiences  with  similar  cases.  Once  classified,  options  are  automatically 
suggested,  based  either  on  standard  operating  procedures  or  on  analogues  that 
have  been  successfully  employed  previously.  Only  in  cases  where  the  initial 
recognition-based  option  is  judged  to  be  unworkable  is  a  second  option 
generated  and  examined  for  feasibility.  This  process  continues  in  a  serial 
fashion  until  a  workable  option  is  found. 

We  have  described  this  process  as  a  Recognition-Pr imed  Decision  (RFD) 
strategy  (Klein,  in  press)  depicted  in  Figure  1.  Three  aspects  of  the  model 
are  of  primary  importance:  serial  versus  concurrent  evaluation,  progressive 
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deepening,  and  the  recognition  of  situations  which  —  we  refer  to  as  situation 
assessment. 

Serial  Evaluation 

The  serial  evaluation  of  options  described  in  the  RPD  model  was  the 
dominant  strategy  found  in  both  of  the  urban  fireground  studies  (Calderwood  et 
al.,  1987;  Klein  et  al.,  1986).  We  wish  to  claim  that  the  RPD  strategy  will 
be  more  prevalent  for  experienced  decision  makers,  relying  as  it  does  on 
memories  for  previously  encountered  similar  events.  Some  support  for  the 
claim  is  suggested  by  the  fact  that  in  the  tank  platoon  study  (Brezovic  et 
al.,  1987),  where  the  platoon  leaders  had  relatively  less  experience  than  any 
of  the  other  decision  makers  studied,  less  than  half  (42%)  of  the  decisions 
were  classified  as  RPD.  Further,  in  the  wildland  study  (Taynor  et  al.,  1987) 
operational  decisions,  with  which  the  coninanders  were  most  experienced,  were 
more  frequently  found  to  be  RPDs  than  were  decisions  involving  organizational 
and  management  problems.  However,  the  Calderwood  et  al.  (1987)  study  of  urban 
fire  ground  commanders  found  no  difference  in  the  frequency  of  RPDs  between 
the  most-experienced  (expert)  and  least-experienced  (novice)  commanders  once 
differences  in  the  number  of  decisions  made  by  individuals  in  these  groups 
were  taken  into  account. 

No  direct  comparisons  of  decision  strategies  along  the  expert-novice 
dimension  were  possible  in  these  studies.  In  each  case,  the  whole  situation 
as  well  as  the  decision  maker  differed.  Because  we  have  held  the  decision 
scenario  constant  in  the  present  study,  we  can  examine  these  issues  under  more 
controlled  conditions.  In  the  present  study  we  sought  support  for  the 
contention  that  concurrent  option  evaluation  is  the  hallmark  of  a  novice  and 
not  the  end  towards  which  decision  makers  should  aspire. 
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gure  1  —  Recognition-Primed  Decision  (RPO)  node! 


Progressive  Deepening 

Standard  approaches  to  beliavioral  decision  theory  assume  that  option 
comparisons  are  carried  out  through  some  form  of  cost-benefit  analysis.  That 
is,  dimensions  are  applied  to  all  options  under  consideration  using  some 
process  to  determine  which  option  is  "best"  on  the  aggregate  of  these 
dimensions  (e.g.,  multi-attribute  utility  theory).  The  serial  evaluation 
process  of  the  RPD  conceptualization  leaves  no  basis  for  such  comparative 
calculations.  Instead,  the  decision  maker  must  somehow  evaluate  the 
"goodness"  of  an  option  in  isolation. 

We  believe  that  the  primary  means  of  evaluating  an  option  involves  a 
process  of  imagination  in  which  the  decision  maker  runs  through  a  mental 
simulation  of  the  outcomes  of  implementing  an  option.  We  have  adopted  the 
term  "progressive  deepening"  to  refer  to  this  evaluation,  a  term  coined  by  de 
Groot  (1965/1978)  to  describe  how  chess  grandmasters  follow  out  a  line  of  play 
to  make  sure  it  does  not  lead  to  blunders.  The  protocols  we  have  collected 
contain  some  vivid  examples  of  these  mental  simulations.  One  example  of  the 
progressive  deepening  strategy  is  an  incident  involving  the  rescue  of  an 
unconscious  woman  who  was  suspended  from  a  highway  overpass.  For  each  type  of 
rescue  harness  the  commander  considered,  he  imagined  how  it  would  be  put  on 
and  how  the  woman’s  back  would  be  supported  once  she  was  free  of  the 
structure.  Several  options  were  rejected  when  the  image  revealed  a  moment 
when  the  risk  would  be  too  great  either  to  the  woman  or  to  his  crew. 

We  suspect  that  this  ability  to  imagine  or  project  a  scene  into  the 
future  is  an  important  component  of  skilled  decision  making.  In  the  expert- 
novice  study  of  urban  firefighting  (Calderwood  et  al.,  1987),  the  Expert  FGCs 
reported  using  imagery  in  over  twice  as  many  decisions  as  did  the  Novice  FGCs 
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(20%  vs.  8%).  A  related  code  classified  each  decision  point  as  to  whether  it 
involved  a  reference  to  a  possible  future  "stato-of-the-world . "  In  this  too, 
twice  as  many  Expert  decision  points  as  Novice  decision  points  (48%  vs.  24% ) 
were  future  oriented. 

In  the  study  of  tank  platoon  leaders  (Brezovic  et  al.,  1987),  a  major 
difference  between  the  protocols  of  the  student  Armored  Officer  Basics  (AOBs) 
and  the  more  experienced  Tank  Crew  Instructors  (TCIs)  was  in  the  reported 
cases  of  what  were  termed  " hypothetical . "  These  were  statements  that 
reflected  consideration  of  future  actions  by  either  platoon  or  enemy  troops. 

In  each  of  16  content  categories,  the  TCIs  had  a  higher  percentage  of 
hypotheticals . 

Again,  the  present  study  provides  an  opportunity  to  produce  converging 
evidence  for  these  findings  in  a  case  in  which  the  situations  being  viewed  are 
held  constant  across  the  decision  makers  and  the  sources  of  biases  present  in 
our  interview  method  are  absent. 

Situation  Assessment 

Behavioral  decision  theory  has  generally  defined  decisions  in  terms  of 
what  Berkeley  and  Humphreys  (1982)  call  ''the  moment  of  choice."  That  is, 
decision  models  describe  how  an  option  is  selected  once  the  relevant  options 
and  evaluation  dimensions  have  been  generated.  It  does  not  generally  try  to 
account  for  the  "pre-decision"  processes  (Gettys,  1983)  involved  in  detecting 
and  structuring  the  decision  problem,  defining  relevant  goals,  and  generating 
plausible  courses  of  actions. 

From  the  RPD  perspective,  however,  processes  prior  to  option  selection 
are  seen  os  critical.  Options  are  evaluated  in  terms  of  the  individual's 
"situation  assessment"  —  the.  understanding  of  a  situation  based  or.  a  sense  of 
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familiarity.  We  have  proposed  that  situation  assessment  entails  at  least  four 
conceptually  distinct  dimensions  of  recognition:  (1)  critical  cues  and  causal 
factors;  (2)  expectancies;  (3)  typical  actions;  and  (4)  plausible  goals.  We 
have  found  it  essential  to  track  each  of  these  dimensions  as  they  cliange  over 
the  course  of  an  incident  in  order  to  understand  the  decision  maker’s  choice 
of  action. 

(Xir  conceptualization  of  situation  assessment  is  related  to  the  concepts 
of  "schema"  (Bartlett,  1932;  Schank,  198G)  and  "mental  model"  (Gentner  & 
Stevens,  1983)  that  are  part  of  many  current  cognitive  theories.  The  general 
notion  is  that  incoming  information  is  categorized,  selected,  edited,  and 
organized  on  the  basis  of  a  person’s  general  knowledge  about  a  domain. 

We  expected  an  analysis  of  the  content  of  protocol  data  obtained  in  this 
study  to  shed  light  on  the  nature  of  situation  assessment  processes. 
Specifically,  we  hoped  to  examine  which  cues  were  being  heeded  at  each 
decision  point,  what  inferences  were  drawn  based  on  the  cues,  and  which  goals 
were  roost  important  in  determining  a  selected  course  of  action.  In  previous 
studies  in  this  program  of  research,  conclusions  about  these  factors  have  been 
limited  by  the  fact  that  there  was  no  way  to  know  what  cues  had  actually  been 
present  in  the  situation  or  what  other  inferences  might  have  been  drawn.  Nor 
has  there  been  any  way  to  compare  how  experts  and  novices  might  differ  in 
their  situational  understanding,  as  each  incident  was  unique  to  the  individual 
reporting  it  to  us. 

'Hie  most  systematic  attempt  to  derive  situation  assessment  categories  in 
this  research  program  was  undertaken  in  the  study  of  tank  platoon  leaders 
(Brezovic  et  al.,  1987).  This  study  had  the  advantage  of  on-site  interviewing 
of  decision  makers  during  force-on- force  field  maneuvers  over  a  three-day 


period.  Interviews  were  carried  out  both  with  the  AOBs  serving  as  platoon 
leaders  and  with  the  TCIs  who  had  several  more  years  of  experience  and  who 
were  responsible  for  evaluating  the  AOBs  during  the  exercises.  Thus,  within 
the  limits  of  physical  proximity,  the  AOBs ,  TCIs,  and  the  interviewers  had  the 
same  information  available. 

}>rotocols  were  analyzed  for  16  situation  assessment  categories  relating 
to  friendly  and  enemy  control,  actions,  training,  support,  and  time 
dimensions.  Hie  contextual  cues  and  areas  of  knowledge  the  AOB3  reported  were 
very  similar  to  the  TCIs,  suggesting  that  differences  were  not  generally  the 
result  of  inattention  to  appropriate  environmental  cues. 

Rather,  the  primary  differences  seemed  to  reside  in  use  of 
"hypotheticals"  —  i.e.,  statements  that  reflected  consideration  of  future 

actions  by  both  platoon  and  enemy  troops.  In  most  of  the  categories 
considered,  the  TCIs  had  more  remarks  coded  as  hypotheticals  than  did  the 
AOBs.  Also  of  interest  was  the  fact  that  for  the  TCIs,  there  were  about  the 
same  number  of  hypotheticals  for  platoon  and  enemy  categories,  whereas  the 
AOBs  were  much  more  focused  on  their  own  platoon’s  movements.  The  AOBs  seemed 
less  able  to  imagine  how  an  enemy  would  react  than  to  anticipate  behaviors  of 
their  own  platoon. 

Results  obtained  in  the  study  of  the  Expert-Novice  fire  ground  decision 
making  (Calderwood  et  al.,  1987)  are  also  relevant  to  this  issue.  This  study 
was  an  initial  attempt  to  use  protocol  analysis  to  capture  the  two  logically 
distinct  processes  of  situation  assessment  and  option  selection.  Each 
deliberated  decision  was  classified  as  having  primarily  involved  deliberation 
about  situation  assessment  (SA-decisions)  or  options  (Option-decisions). 

Ihese  dimensions  correspond  to  the  operational  distinction  between  the 


1-9 


questions  "What  is  my  situation?"  and  "What  am  I  going  to  do  about  it?"  We 
hypothesized  that  the  expert  decision  makers  would  make  relatively  more  SA- 
decisions  than  novices,  in  keeping  with  findings  in  related  fields.  For 
example,  Larkin  (1981)  found  that  expert  physics  problem  solvers  expend  more 
effort  in  constructing  some  kind  of  analogue  to  a  physical  representation 
before  starting  to  solve  the  problem,  whereas  novices  are  more  likely  to 
proceed  almost  immediately  to  setting  up  equations.  Similarly,  Sternberg 
(1986)  found  that  more  intelligent  problem  solvers  tended  to  put  more  of  their 
time  in  the  encoding  of  an  analogy  problem  and  less  time  operating  on  these 
encodings . 

The  results  of  this  study  can  only  be  viewed  as  suggestive  given  the 
previously  described  difficulty  of  directly  comparing  decisions  made  by  the 
expert  and  novice  participants.  Nonetheless,  when  the  percents  of  SA- 
decisions  and  Option-decisions  for  each  incident  protocol  were  computed, 
experts  had  an  equivalent  percent  of  each  type  ( 30%  and  3 OX)  whereas  novices 
had  a  higher  percent  of  Option  decisions  (29X)  than  of  SA  decisions  (18X).1 
Thus,  the  pattern  of  these  relative  percentages  were  in  the  predicted 
direction. 

Again,  the  present  study  enables  the  examination  of  situation  assessment 
by  examining  the  content  of  decision  protocols  generated  by  more-  and  less- 
experienced  comnanders  in  response  to  the  same  events. 

This  study  represents  a  significant  departure  from  previous  studies  in 
this  series.  The  previous  investigations  relied  on  retrospective  interviews 

1 Note  that  the  remaining  percentages  reflect  the  frequency  of 
"Automatic"  or  non-deli be rated  decisions  for  which  the  SA  versus  Option 
distinction  was  not  made. 
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to  probe  for  information.  Hie  present  study  obtained  think -aloud  protocols 
during  an  incident  by  stopping  the  action  at  pre-selected  "probe  points." 

This  necessitated  creating  a  simulated  task  environment  in  which  we  could 
control  the  information  being  received  by  the  decision  maker. 

Considerable  effort  was  expended  in  this  project  developing  the  simulation 
materials  and  protocol  analysis  methods. 

Hie  study  was  designed  to  address  several  inter-related  issues  of 
relevance  to  RPD  model  development: 

♦Does  this  alternate  method  provide  convergent  evidence  for  the  serial 
evaluation  strategy  described  by  the  RPD  model? 

*  Does  the  method  provide  a  technique  for  examining  progressive  deepening 
and  imagery  as  a  means  of  option  evaluation? 

*  What  aspects  of  situation  assessment  are  spontaneously  reported  — 
wtiat  cues,  inferences,  and  goals  are  associated  with  command  decisions? 

*  To  what  extent  are  these  factors  associated  with  domain  expertise? 
Development  of  Urban  Fire  Ground  Command  Simulations  (FGC-Simulation) 

In  designing  a  simulation  format,  we  faced  an  initial  dilemma:  how  to 
realistically  engage  a  commander  in  the  decision  making  process  while  at  the 
same  time  preserving  control  over  the  input  features  of  an  incident  so  as  to 
allow  comparisons  across  commanders,  This  problem  exists  because  decisions  in 
this  environment  occur  in  the  context  of  action  sequences,  where  the  outcome 
of  any  particular  decision  affects  the  subsequent  course  of  events.  One 

approach  to  this  dilemma  has  been  interactive  videodisc  and  computer 
presentation  technologies  that  allow  action  sequences  to  be  played  out  along 
some  pre-specif ied  number  of  alternative  "branches."  Although  this  approach 
allows  repetition  and  comparison  of  scenarios  within  the  limits  of  the 
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specified  branches)  the  supporting  technology  tends  to  be  very  expensive  both 
to  develop  and  to  run.  More  importantly,  it  assumes  that  the  selected 
branches  accurately  represent  and  effectively  exhaust  the  natural  response 
categories  of  the  decision  maker. 

Our  solution  was  to  develop  scenarios  that  were  organized  around 
predefined  decision  points  (as  opposed  to  options)  that  represented  key  events 
within  the  overall  incident.  The  scenario  is  interrupted  at  each  of  these 
decision  points  and  verbal  protocol  data  obtained.  Once  the  participant  has 
responded,  the  scenario  is  restarted  and  the  narrator  provides  information 
about  the  actions  that  were  actually  taken  at  that  point  by  the  "real”  FGC  who 
was  in  charge  of  the  incident.  This  device  serves  to  reorient  each 
participant  to  a  cormnon  set  of  circumstances  before  allowing  the  incident  to 
develop  further. 

Scenarios  were  based  on  interviews  with  PGCs  obtained  in  a  previous  study 
(Calderwood  et  al . ,  1987).  Events  were  recreated  using  an  audio-visual  format 
to  present  the  details  of  the  incident  from  the  commander’s  perspective.  The 
simulation  presents  relevant  radio  communication  and  a  series  of  graphic 
slides  of  an  incident  from  the  time  of  the  initial  alarm  to  a  point  where  the 
incident  has  been  brought  under  control.  All  events  are  depicted  from  the 
point  of  view  of  the  FGC.  A  narrator  supplies  needed  background  information 
that  would  be  known  to  the  conmander  or  would  become  available  in  other  ways 
during  an  actual  incident.  Key  events  are  portrayed  in  near  real-time. 

We  were  fortunate  to  iiave  a  pool  of  incident  accounts  on  which  to  base 
the  scenario  scripts.  Nonetheless,  the  process  of  selection  and  development 
presented  several  challenges,  for  we  wanted  to  depict  the  look  and  sound  of 
the  fi reground  as  completely  and  accurately  as  possible.  In  choosing 
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incidents  that  would  be  suitable  for  simulation  development,  we  adopted  the 
following  criteria: 

Complexity.  We  wanted  to  retain  the  dynamic,  complex  nature  of  decision 
making  on  the  fireground.  This  argued  for  using  incidents  that  involved  a 
series  of  decisions  made  in  response  to  shifts  in  situational  elements,  and  a 
variety  of  cues  present  in  the  situation. 

Cue  availability.  Given  our  interest  in  situation  assessment  processes, 
presence  of  critical  cues  that  were  immediately  available  to  the  comnander  was 
another  salient  dimension.  (Xir  emphasis  here  was  on  cues  that  could  be 
depicted  without  our  prestructuring  or  interpreting  them  for  the  participant. 
This  meant  that  the  majority  of  critical  cues  in  an  incident  had  to  be  visual 
or  auditory,  because  of  the  difficulty  of  representing  olefactory  or  tactile 
cues  in  a  simulation. 

Incident  type.  We  warn  ted  the  incidents  to  represent  a  range  of  issues 
and  of  types  of  tactical  and  strategic  decisions,  At  the  same  time,  we  had 
teen  cautioned  by  trainers  and  others  in  the  fire  service,  that  fireground 
simulations  are  sometimes  devised  that  are  so  "far  out"  that  they  are  not 
believable.  If  the  simulations  appeared  full  of  tricks  or  too  much  like  a 
game,  we  feared  that  the  participants’  engagement  would  suffer.  Thus,  the 
incidents  had  to  be  believable  and  to  represent  the  normal  range  of  decision 
making  for  officers  in  this  geographic  area. 

We  began  by  developing  script  outlines  and  storyboards  of  the 
accompanying  graphics  for  seven  incidents.  Working  with  an  experienced 
officer/trainer,  we  selected  two  scenarios  for  full  development  and  a  third 
that  could  function  as  a  practice.  The  incidents  were  judged  to  present 
different  types  of  command  challenges.  The  two  study  simulations  were  both 
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structure  fires,  one  at  an  occupied  apartment  building  and  one  at  an 
unoccupied  restaurant  t  a  large  historical  building.  The  practice  scenario 
involved  an  overturned  tanker  truck  on  a  highway. 

Development  oi  the  final  scenarios  was  an  iterative  process.  Graphic 
artists  were  retained  to  draw  the  structures  and  depict  visual  cues  of  the 
fire  and  surrounding  area  at  several  key  points.  A  sound  engineer  oversaw 
development  of  the  audio  portion  of  the  simulation.  Radio  communication  was 
supplied  by  recording  voices  speaking  through  two-way  radios.  A  professional 
actress  was  retained  to  narrate  the  incidents.  Background  noise  and  sound 
effects  appropriate  to  the  fireground  were  added.  At  each  step  in  this 
process,  we  sought  feedback  about  the  realism  of  the  representations  from  the 
)'XjC  consultant. 

We  piloted  the  simulated  incidents  with  four  firefighters,  two  of  whom 
are  highly  experienced  PGCs.  Final  revisions  and  corrections  were  made  on  the 
basis  of  their  comments  and  the  graphics  were  then  converted  to  slides. 

Method 

Study  Participants 

Participants  were  all  professional  firefighters  employed  by  the  City  of 
Dayton  Fire  Department ,  Dayton,  OH.  We  worked  with  the  Department’s  Chief 
Suppression  Officer  to  recruit  volunteers  who  represented  a  range  of  command 
experience.  Because  the  scenarios  required  the  participant  to  take  charge  of 
a  major  incident,  it  was  not  feasible  to  use  new  firefighters  or  firefighters 
with  virtually  no  coumand  experience.  The  22  FGCs  who  comprised  the  final 
sample  are  seasoned  firefighters  who  were  judged  to  vary  considerably  in 
command  experience  and  skill  by  their  chief.  Expert  PGCs  had  an  average  of 
18.9  total  years  (range  =  10-27  years)  and  Novices  had  an  average  of  13.5 
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total  years  of  firefighting  experience  (range  =  7-23  years).  Experts  had  an 
average  of  11.0  years  in  command  positions  (range  =  6-20  years),  as  opposed  to 
an  average  of  4.4  years  of  coom&nd  experience  in  the  Novice  group  (range  =  1- 
12  years) . 

Procedure 

In  the  final  form  each  FGC-simulation  scenario  contained  multiple 
decision  points  that  span  the  duration  of  the  incident.  The  scenario 
involving  the  overturned  tanker  truck  was  used  in  the  present  study  for 
practice.  It  contains  three  decision  points  and  two  graphic  depictions.  The 
apartment- fire  scenario  contains  five  decision  points,  three  graphic 
depictions  of  the  scene,  and  two  overhead  maps  that  show  apparatus  and  hydrant 
placement.  The  Inn-fire  scenario  contains  six  decision  points,  three 
graphics,  and  one  overhead  map. 

Hie  simulated  incidents  were  presented  using  a  tabletop  audio/slide 
projector.  Sessions  were  conducted  individually  by  one  of  two  trained 
examiners  who  were  blind  to  rank  or  expertise  classification. 

After  a  brief  introduction,  the  simulation  format  was  explained. 
Participants  were  asked  to  imagine  themselves  in  the  conm&nd  role  in  each 
incident,  and  told  that  at  certain  points  the  tape/slide  presentation  would  be 
interrupted.  They  were  instructed  that  when  this  occurred,  they  should: 

"...  say  aloud  all  the  things  you  are  thinking  to  yourself,  and 
even  to  describe  any  images  or  memories  that  come  to  mind.  Although 
we  are  interested  in  your  plans  and  any  actions  you  might  take,  we 
are  also  interested  in  what  you  are  hoping  to  accomplish  with  your 
actions,  what  you  are  noticing,  and  any  other  options  you  are 


1-15 


considering.  It  is  safe  to  say  that  we  are  interested  in  anything 
that  pops  into  your  head." 

It  was  explained  that  once  they  had  told  us  what  they  were  thinking,  the 
tape  would  be  restarted  and  would  tell  them  what  the  FGC  actually  did  in  this 
incident.  We  noted  that  some  of  the  actions/decision  depicted  might  be 
considered  controversial,  and  that  they  might  not  always  agree  with  what  the 
FQC  did.  In  that  event,  we  asked  that  they  save  criticism  until  the  end  of 
each  incident,  when  they  would  have  a  chance  to  comment.  It  was  thought  that 
providing  an  opportunity  to  critique  the  incident  at  the  end  would  foster 
additional  engagement,  and  keep  participants  from  becoming  sidetracked  in 
disagreements  about  how  the  incident  was  being  handled. 

lhe  participants  were  told  that  the  initial  scenario  was  for  practice,  to 
give  them  an  idea  of  the  format  and  answer  any  questions.  After  the  practice, 
they  were  reminded  that  they  were  the  commander  throughout  the  incidents,  and 
that  they  should  say  "whatever  comes  into  your  head."  After  answering  any 
questions,  we  presented  the  remaining  two  scenarios.  Verbal  protocol  data  in 
response  to  all  three  scenarios  was  recorded  on  a  separate  audiotape,  once 
permission  to  tape  record  had  been  obtained.  No  participant  declined  to  be 
recorded. 

At  each  predefined  decision  point,  the  audio/projector  was  stopped  and 
the  examiner  asked:  "What  ore  you  thinking  about  at  this  point?"  At  the  end 
of  the  participant’s  verbal  response,  the  examiner  asked:  "Anything  else?" 
Participants  were  not  prompted  in  any  other  way.  When  the  participant 
declined  further  conroent,  the  examiner  restarted  the  tape.  Participants 
occasionally  raised  questions  about  aspects  of  the  simulation  or  the  incident 
itself.  If  the  request  was  for  information  contained  in  the  taped  narration, 
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or  available  on  the  screen,  examiners  would  respond  with  the  requested 
information.  If,  however,  the  request  was  for  additional  information  not 
contained  in  the  scenario,  the  examiners  politely  declined.  This  was  done  to 
ensure  comparability  across  participants . 

Biographical  information  (e.g.,  years  of  firefighting  experience,  years 
of  command)  was  obtained  at  the  end  of  each  session  when  data  gathering  was 
c'--Tiplete .  Sessions  took  from  45  minutes  to  2  hours,  depending  on  the  amount 
of  talk. 

Protocol  Analysis 

Coding  Procedures 

All  of  the  utterances  produced  during  the  probe  points  for  the  three 
incidents  were  transcribed.  All  of  the  speech  recorded  for  each  subject  for  a 
particular  probe  point  within  an  incident  constitutes  a  protocol .  Each 
protocol  was  segmented  by  the  transcriber  into  paragraphs  reflecting  naturally 
occurring  pauses  in  the  speech.  Protocol  transcriptions  were  keyed  to  a 
subject  number  but  did  not  indicate  the  experience  level  or  name  of  the 
participant. 

A  sampling  of  protocols  from  each  probe  point  was  read  prior  to  beginning 
formal  coding  procedures  and  three  probe  points  were  selected  as  a  sample  of 
the  larger  set  of  14  probe  points:  the  first  and  fifth  probe  points  in  the 
apartment  fire  incident  (Probe  points  1.1  and  1.5)  and  the  first  probe  point 
of  the  Inn  incident  (Probe  point  2.1).  The  first  probe  points  in  each 
incident  contained  more  words  (and  presumably  more  remarks)  than  later  probes 
and  would  establish  a  context  for  interpreting  later  probe  points.  The  single 
later  probe  point  (1.5)  was  chosen  arbitrarily. 
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Remark  and  category  definition.  The  first  step  of  the  protocol  analysis 


was  to  define  the  basic  units  of  analyses.  The  goal  was  to  segment  the 
connected  discourse  into  "chunks"  at  a  meaningful  level  for  addressing 
relevant  research  questions.  This  might  have  been  anything  from  single  words 
to  short  phrases  to  higher-level  topics  abstracted  from  groups  of  sentences. 

We  chose  as  our  unit  of  analysis  the  remark.  which  we  defined  as  a  word, 
phrase ,  sentence,  or  group  of  sentences  that  have  a  coninon  topic  or  referent. 
The  list  of  remark  types  constituting  the  coding  categories  are  described 
tielow.  Thus,  remarks  are  somewhat  circularly  defined  in  relation  to  the 
content  categories  we  used  to  address  the  research  questions  of  interest. 

Every  remark  contained  in  a  protocol  was  classified  as  belonging  to  one 
and  only  one  of  the  following  categories: 

1.  CUE  -  (type).  Remarks  that  express  an  awareness  of  the  information 
provided  by  the  scenario.  They  are  the  facts  of  the  present  case  as  viewed  by 
the  subject  (Example:  "I  can  see  the  fires  in  the  basement).  The  cue  "type" 
is  a  summary  of  the  information  content  —  what  was  noticed. 

2.  CUE-Deliberation  -  Remarks  that  express  uncertainty  about  the  meaning  of 
a  cue  or  set  of  cues  (e.g.,  "This  could  be  a  ...")  indicating  a  need  to 
deliberate  or  come  to  a  decision  about  the  current  state-of-the-world. 

3.  CUE-Anticipation  -  Remarks  that  involve  a  prediction  about  an  anticipated 
future  state-of-the-world  based  on  present  cues  or  inferences.  (Example: 

"From  the  looks  of  it,  fire’s  going  to  run  that  wall  right  into  the  attic.") 

4.  KNOWLEDGE  -  Remarks  that  express  domain-relevant  knowledge  of  fire  ground 
factors.  TTiese  may  occur  in  assessing  the  meaning  of  cues  in  the  present  case 
or  to  evaluate  or  generate  an  action  plan.  General  knowledge  may  be  about 
equipment,  resources,  crew  functions,  building  structure,  or  fire  dynamics, 
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etc.  (Example:  "Balloon  construction  means  that  a  fire  is  likely  to  spread 
vertically  very  rapidly;"  "Blackish  smoke  indicates  the  presence  of  a 
hazardous  material.")  This  is  book learning  that  is  applied,  modified,  and 
interpreted  in  assessing  cues  in  the  specific  incident. 

Knowledge  expressed  as  standard  operating  procedures  (SOP)  were 
considered  as  a  special  case  of  knowledge.  These  remarks  reflect  standard 
strategy  and  tactics  on  the  fireground  (example:  "Life  is  the  most 
important"),  or  about  procedures  that  are  considered  standard  by  this 
department  in  particular.  (Example:  "Our  second-in  engine  functions  as  the 
supply,"  "trucks  have  priority  for  the  front  of  the  building"). 

5.  ACTION  -  (type).  Remarks  that  express  the  current  actions  or  plans  that 
the  PGC  will  implement. 

6.  ACTION  Deliberation  -  Remarks  that  express  uncertainty  about  the  fiction 

(Example:  "I  could/might  do  ...")  indicating  the  need  for  deliberation  or 

further  evaluation  before  the  action  is  implemented. 

V .  CONTINGENCY  -  Action  remarks  that  indicate  that  a  plan  would  only  be  put 
into  effect  when  or  i£  a  future  condition  is  met.  (Example:  "If  it  gets  to 
the  attic  and  mushrooms,  I’ll  pull  the  crews  out  and  go  to  master  streams.") 

8.  GOAL  -  (type)  -  explicit  statements  about  the  purpose  or  reason  for 
taking  an  action.  These  specify  what  the  PGC  hopes  to  accomplish.  (Example: 
"We  need  to  get  that  fire  vented  so  we  can  clear  the  smoke  and  heat  out  of 
there.") 

9.  REMINDING  -  Remarks  that  refer  to  prior  experiences  that  the  simulation 
has  brought  to  mind.  (Example:  "We’ve  got  a  structure  a  lot  like  this  one 
right  up  here  on  Main  Street.") 
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10.  APPRAISAL  -  Remarks  that  reflect  an  evaluation  of  "How  am  I  (as  PGC) 
doing?"  or  "How  will  this  all  turn  out?"  (Example:  "This  is  a  tough  fire,  I 
think  we'll  lose  the  building.") 

11.  META-O0GNIT1ONS  -  Remarks  that  express  how  the  FGC  would  go  about 
thinking,  making  decisions,  etc.  (Example:  "I’ll  be  keeping  in  mind  that  the 
crews  will  wear  out  fast  in  this  heat  —  gotta  keep  an  eye  on  that.") 

12.  CRITIQUES/QUESTIONS  -  comments  or  questions  about  the  simulation  itself 
or  the  simulation  PGC's  handling  of  the  incident. 

13.  MISCELLANEOUS  -  incomplete  or  indiscernible  remarks  or  remarks  that  do 
not  fit  any  of  the  study  categories. 

A  sample  coded  protocol  is  presented  in  Table  1 . 

Coding  was  done  for  all  the  protocols  in  a  given  probe  point  before  going 
on  to  the  next.  We  found  that  it  greatly  facilitated  coding  to  have  the 
context  of  a  probe  point  clearly  in  mind,  especially  in  designating  the  cue 
and  action  types  being  expressed.  It  also  meant  that  the  data  for  a 
]»articular  probe  point  might  be  discarded  and  the  protocols  re-coded  if 
reliability  for  that  point  did  nut  reach  acceptable  levels.  Inter-coder 
reliability  was  assessed  periodically  throughout  the  coding. 

Three  coders  (the  authors)  were  randomly  assigned  protocols  from  each 
probe  point,  although  not  all  coders  coded  the  same  number  of  protocols. 

Coding  Reliability 

After  developing  the  coding  definitions  and  procedures ,  several  protocols 
were  coded  by  each  of  the  three  coders  (the  authors)  in  order  to  test  and 
refine  our  understanding  of  the  coding  categories.  Formal  assessment  of 
inter-coder  reliability  was  carried  out  concurrently  with  the  coding  of  the 
three  decision  points  coded  for  the  present  study.  This  was  done  in  the 
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TABLE  1 


Sample  -  Coded  Protocol 

1  Cue  [’Okay,  I  can  see  that  we  have  a  fairly  large 

2  Cue  structure,]  [2fire  on  the  second  floor;] 

a  Cue  [3dcesn’t  appear  that  the  first  floor  is 

<  Cue  involved  at  this  point.]  [4It  appears  that 

s  Cue  engine  1  laid  out  coming  in,  they  laid  out 

8  Knowledge:SOP  their  supply  ocming  in,]  [$I  don’t  see  where 

there  is  an  attack  line  down  yet.]  But, 

(#  initial  companies,  that  would  be  their  first 
response,  to  go  ahead  and  lay  the  initial 
attack  line,  make  entry  into  the  building, 
make  the  stairwell  to  the  second  floor,  and 
start  checking  it  for  the  fire.) 


7  Cue 
0  Action 
•  Action 

10  Knowledge :Fi reground 
factors 


[7 Since  this  is  a  relatively  old  building,] 
[•ry  first  response  would  be  to  go  ahead  and 
stage  a  second  alarm,]  [9I  would  need  DP&L, 
it  is  more  than  likely  they  are  a  large 
natural  gas  consumer.] 


1 1  Action 

12  Appraisal 

13  Cue 
'*  Cue 


C1 1 1  would  go  ahead  and  stage  a  second  medic,] 
[12this  is  going  to  be  a  tough  fire]  and  [,3we 
may  start  running  into  heat  exhaustion 
problems;]  [,4it  is  70  degrees  now  and  it  is 
going  to  get  hotter.] 


15  Action 
18  Action 

17  Action-Elaboration 
13  Goal 


[,5The  first  truck,  I  would  go  ahead  and  ►'X.- 
them  open  the  roof  up]  t-xJ  t1 8  the  second  truck 
I  would  go  ahead  send  them  on  inside]  and 
[ 1 7 have  them  start  ventilating,  start  knocking 
the  windows  out  and  working  with  the  initial 
engine  crew,  false  ceilings  and  get  the  walls 
opened  up.)  [1BQet  to  the  source  of  the  fire, 
get  it  knocked  down.) 


1  •  Cue 

20  Action 

21  Anticipation 

22  Action 

23  Cue-Deliberation 

24  Anticipation 
23  Metacognition 
28  Contingency 

27  Anticipation 

28  Contingency 

28  Action-Deliberation 
30  Action 


[18I  am  ass’jning  at  this  point  I  do  not  have 
any  other  engines  on  the  scene.]  [20As  soon  as 
I  can,  go  ahead  and  order  the  second  engine  to 
hook  up  to  the  supply  and  ptirp  to  engine  1,) 
[2,I  am  assuning  engine  2  will  probably  be 
there  in  a  second]  and  [2  2  have  them  pcnp  to 
engine  1,  supply  them.]  [23I  don’t  knew  how 
long  the  supply  line  lay  is,]  [24but  it 
appears  we  are  probably  going  to  need  more 
water  than  one  supply  line  is  going  to  give 
us. J  [25So  I  would  keep  in  mind,]  [29unless  we 
can  check  the  fire  fairly  rapidly,]  [27that  we 
are  going  to  have  to  have  seme  more  water,] 

[2 8 so  start  thinking  of  other  water  sources.] 
[2* Consider  laying  another  supply  line  to 
engine  1]  and  [30 stand  back  and  watch.] 
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interest  of  efficiency  and  also  as  a  check  on  any  drift  in  our  agreement  over 
time. 

Two  separate  aspects  of  reliability  were  assessed  —  inter-coder 
agreement  in  segmenting  the  protocols  into  remark  units,  and  inter-coder 
agreement  in  classifying  the  remark  segments  into  coding  categories,  Ihe 
first  task,  assessing  the  reliability  of  remark  segmenting,  is  difficult 
because  differences  between  a  match  and  a  mismatch  are  usually  a  matter  of 
degree  of  agreement.  That  is,  a  difference  in  any  given  segment  can  carry 
over  to  several  subsequent  segments.  The  second  task,  assessing  the 
reliability  of  remark  classifications,  is  interdependent  with  the  first  task. 
It  is  hard  to  classify  a  "thing"  the  same  way  if  there  is  no  agreement  about 
which  "thing"  is  being  classified.  In  related  research,  these  problems  are 
sometimes  bypassed  by  having  a  single  criterion  coder  responsible  for 
segmenting  protocols,  or  by  only  computing  classification  reliability  on 
remarks  that  were  segmented  with  good  agreement.  Both  of  these  procedures 
would  appear  to  inflate  the  degree  of  agreement  among  coders  starting  from  an 
unstructured  verbal  protocol. 

Our  solution  was  to  adopt  a  sampling  strategy  that  eliminated  the 
interdependence  of  these  two  aspects  of  coding.  Because  we  used  a  word- 
processor  to  print  the  columns  of  transcribed  text,  the  text  was  broken 
arbitrarily  at  the  end  of  a  line.  We  chose  the  first  word  of  every  third  line 
as  an  anchor  on  which  to  compare  coders.  A  subset  of  protocols  from  each 
probe  point  were  chosen  at  random  and  coded  by  at  least  two  coders.  Six 
protocols  for  probe  point  1.1  were  coded  by  all  three  coders.  To  increase 
efficiency,  only  partial  overlap  between  coders  was  carried  out  for  probe 
points  1.5  and  2.1.  Coder  C  coded  only  two  protocols  on  which  to  assess 
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reliability  from  each  of  the  two  remaining  probe  points.  Coders  A  and  B  were 
responsible  for  coding  the  bulk  of  the  protocols  and  they  overlapped  on  five 
protocols  for  probe  point  1.5  and  seven  protocols  for  2.1. 

Reliability  of  remark  segmenting  was  assessed  by  comparing  segments 
containing  each  anchor  word  on  a  three  point  scale.  High  match  indicated 
almost  perfect  agreement  (ignoring  prepositions  or  articles)  in  designating 
both  the  beginning  and  ending  of  a  remark  segment.  Medium  match  indicated  a 
good  degree  of  overlap  in  the  segments  —  these  segments  either  started  or 
ended  in  the  same  place  on  the  transcript.  Low  match  indicated  discrepant 
segmenting  of  remarks.  That  is,  coders’  remark  segments  neither  began  nor 
ended  similarly.  Percent  of  remarks  at  each  level  of  match,  for  each  coder 
pair,  are  presented  in  Table  2.  These  data  indicate  a  high  level  of  agreement 
among  the  three  coders  in  remark  segmenting. 

Inter-coder  agreement  on  classification  of  remarks  into  content 
categories  was  assessed  using  the  kappa  coefficient  (Cohen,  1960).  Kappa  is  a 
chance-corrected  measure  of  nominal  scale  agreement,  first  developed  to  assess 


Table  2 

Percent  Agreement  for  Remark  Segments  for  Coders  A,  B,  and  C 


Level  of  Match 
High 
Medium 
Low 

•Based  on  167  remarks. 
b Based  on  204  remarks. 
c Based  on  119  remarks. 


with  B*  B  with 

69.8  64.6 

28.6  34.1 

1.5  1.3 


£b  C  with  Ac 

65.5 
33.7 
0.9 
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reliability  of  patient  assignment  to  medical  diagnostic  categories.  It 
provides  a  more  conservative  measure  of  inter-coder  reliability  than  do  simple 
percent  of  agreement  measures  which  often  provide  inflated  indices  of  coding 
reliability  (Fleiss,  1981).  Kappa  coefficients  for  major  coding  categories 
for  coder  pairs  AB  and  BC  are  presented  in  Table  3. 2  Several  coding 
categories  occurred  so  infrequently  that  their  reliability  could  not  be 
assessed.  In  these  cases,  data  were  either  combined  into  a  higher-level 
coding  category  (e.g.  "Anticipation"  was  collapsed  into  the  "Cue"  category)  or 
were  dropped  altogether. 


Table  3 

Reliability  of  Remark  Category  Coding:  Kappa  Coefficients 
Category  A  with  B*  B  with  Cb 


Die 

.52 

.56 

Knowledge:  Fi reground  factors 

.26 

.51 

Knowledge  (SOP) 

.60 

.55 

Action 

.68 

.69 

Contingency 

.58 

.23 

Coals 

.58 

.73 

All  Categories 

.54 

.50 

•Based  on  174  remarks 
b Based  on  111  remarks 


2  In  the  interests  of  efficiency,  all  three  coders  did  not  code  all 
protocols.  The  number  of  protocols  that  coders  A  and  C  coded  in  common  was 
judged  too  small  to  allow  good  measurement  of  their  coding  reliability. 
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In  general,  kappa  values  that  exceed  .75  are  considered  excellent 
agreement  beyond  chance,  kappas  between  .40  and  .75  indicate  fair  to  good 
agreement  beyond  chance  and  those  below  .40  are  considered  poor.  Based  upon 
these  criteria,  levels  of  inter-coder  agreement  in  the  present  study  are 
generally  quite  good.  As  a  check  on  variation  in  reliability  across  probes, 
overall  kappa  coefficients  were  computed  separately  for  probes  1.1  and  2.1. 
These  values  were  very  close:  .61  for  probe  point  1.1  and  .57  for  probe  point 
2.1. 

Results 

One  concern  in  a  study  of  this  type  is  how  well  participants  respond  to 
the  simulation  format  and  whether  they  are  able  to  verbalize  their  thinking. 
Participants  generally  reported  that  they  found  the  task  interesting  and  they 
seemed  to  have  little  trouble  talking  during  the  probe  point  pauses.  The 
number  of  words  spoken  during  the  selected  probe  points  ranged  from  39  to  723. 
Averages  for  the  three  selected  probe  points  were  337  words  for  probe  point 
1.1,  166  words  for  probe  point  1.5  and  289  words  for  probe  point  2.1. 

Example  protocols  from  two  Experts  and  two  Novices  from  probe  point  1.1  are 
included  as  Appendix  A. 

Analysis  of  Category  Frequencies 

Remarks  contained  approximately  13  words  on  the  average.  As  expected 
from  the  word  count,  the  first  probe  points  (1.1  and  2.1)  contained  more 
remarks  on  the  average  (24.95  and  22.85,  respectively)  than  did  probe  point 
1.5  (13.75).  The  average  number  of  remarks  for  the  Experts  and  Novices  for 
each  of  the  selected  probe  points  is  shown  in  Table  4.  There  do  not  appear  to 
be  large  or  systematic  differences  in  the  number  of  remarks  for  these 
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Table  4 


Average  Number  of  Remarks  for  Experts  and  Novices 
for  Three  Probe  Points 


Probe  Point 

Novices 

1.1 

26.27 

22.18 

1.5 

14.09 

13.45 

2.1 

22.36 

23.27 

groups;  nonetheless,  the  protocol  frequencies  are  expressed  as  a  percent  of 
the  remarks  in  the  protocol  in  order  to  equate  the  conditions. 

The  average  percentages  of  11  remark  categories  for  the  Experts  and 
Novices  are  shown  in  Table  5.  Several  of  the  coding  categories  will  not  enter 
into  this  interpretative  analysis,  although  they  were  retained  in  the  data 
pool  for  purposes  of  obtaining  overall  remark  percentages.  For  example, 
across  probe  points,  repetitions  of  previous  remarks  comprised  7. IX  of  the 
coded  remarks.  We  felt  that  including  these  in  the  analysis  was  not 
informative  enough  to  justify  the  added  complexity.  The  Critique/^iestion 
category  comprised  another  3.4X  of  the  remarks.  These  are  discounted  because 
they  are  irrelevant  to  the  decision-making  issues  being  addressed.  The 
Appraisals,  Remindings,  and  Meta-Cognition  categories  comprised  less  than  2% 
of  remarks  in  each  probe  point.  These  categories  do  not  enter  into  the 
present  discussion,  although  we  did  examine  each  of  the  examples  of  these 
categories  for  clues  they  might  contain  about  decision-making  processes  (these 
exasmples  are  discussed  in  a  later  section).  Finally, 
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Average  Percent*  of  Decision  Remark  Categories 
for  Experts  and  Novices  for  Three  Probe  Points  (N=22) 
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itages  are  proportion  of  all  remarks  contained  in  a  protocol.  Tabled  findings  do  not  sub  to  100%. 


remarks  coded  Miscellaneous  comprised  only  4.6%  of  the  coded  remarks ,  and 
these  were  also  dropped  to  simplify  the  analysis. 

We  know  of  no  generally  accepted  methods  for  establishing  statistical 
significance  for  data  such  as  these.  We  adopted  the  following  criteria  for 
determining  which  of  the  observed  differences  between  Experts  and  Novices 
should  be  considered  "meaningful:" 

1}  The  absolute  difference  between  the  category  means  is  greater  than 
the  pooled  estimates  of  the  standard  deviation  of  the  protocol  percentages  for 
the  category  within  that  probe  point  (Large  Difference  support). 

2)  The  direction  of  the  differences  between  Experts  and  Novices  for  a 
category  is  consistent  across  the  three  probe  points  (Replication  support). 
These  criteria  seemed  to  provide  a  reasonable  balance  between  the  costs  of 
Type  I  and  Type  II  errors  for  this  exploratory  analysis. 

Table  5  is  organized  into  groupings  corresponding  to  the  production-rule 
form  suggested  by  the  RPD  model:  if  CONDITION,  then  do  ACTION.  In  the  RPD 
model,  the  CONDITION  determination  is  termed  a  situation  assessment.  The 
situation  assessment  is  based  on  incoming  information  and  case-relevant 
knowledge.  If  there  is  uncertainty  with  regards  to  the  CONDITION,  then 
deliberation  must  occur  to  determine  what  condition  is  most  appropriate. 
Deliberation  may  involve  seeking  more  information,  thinking  about  alternative 
interpretations  of  the  cues,  or  anticipating  future  developments  by  projecting 
a  scene  forward  in  time.  Situation  assessment  should  be  reflected  in  the  Cue 
and  Knowledge  categories. 

It  became  clear  in  coding  the  protocols  that  the  Cue  and  Knowledge 
categories  really  represented  a  continuum  of  inferential  interpretation.  For 
example,  remarks  in  probe  point  2.1  frequently  referred  to  the  fact  that  the 
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building  was  old.  This  remark  was  coded  as  CUE  -  (type  =  structure  is  old), 
'ihis  information  had  been  provided  as  part  of  the  background  of  the  case,  it 
required  no  inference.  However,  in  a  real  incident  a  building's  age  may  have 
to  be  inferred  based  on  an  interpretation  of  other  cues.  Other  remarks 
concerned  which  crews  or  equipment  were  available,  such  as  "I  have  an  extra 
engine  available."  Such  remarks  were  frequently  so  matter-of-fact  that  one 
coder  tended  to  classify  them  as  Cues.  Another  coder  tended  to  classify  these 
remarks  as  Knowledge,  because  a  judgment  of  "availability"  required  knowledge 
of  the  relationship  between  resources  and  needs.  This  ambiguity  accounts  for 
the  low  inter-coder  agreement  for  the  Knowledge  category  (kappa  =  .26)  for 
coders  A  and  B.  When  these  categories  are  combined,  agreement  is  raised  into 
the  "good"  range  (kappa  =  .57). 

A  higher  overall  percent  of  Experts’  remarks  were  classified  as  Situation 
Assessment  remarks  than  were  Novices’  (Expert  Situation  Assessment  =  41. 8X, 
Novice  Situation  Assessment  =  31. IX)  and  the  direction  of  this  difference  is 
supported  by  the  Replication  criterion.  However,  these  differences  are  not 
large  and  inspection  of  Table  5  reveals  that  they  are  primarily  related  to  the 
Knowledge  and  Cue -deliberation  remarks.  The  fact  that  these  categories  are 
designed  to  reflect  inferential  and  reasoning  processes,  provides  support  for 
the  view  of  expertise  on  which  the  RPD  model  is  based.  That  is,  to  the  extent 
that  remark  categories  reflect  relative  "amounts"  of  processing,  the  Experts 
appear  to  deliberate  more  frequently  about  the  nature  of  the  situation  than  do 
Novices.  Said  differently,  the  consistently  higher  percentage  of  Situation 
Assessment  remarks  support  the  notion  that  Experts  expend  more  effort  in 
building  an  accurate  "mental  model"  of  a  situation  on  which  to  base  decisions 
about  what  actions  to  take.  The  difference  found  in  the  Cue-deliberation 
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category  for  probe  point  2.1  will  be  discussed  in  the  next  section  in  which 
the  content  of  the  category  remarks  is  examined. 

Examination  of  the  Action  Assessment  categories  reveals  that  Novices  have 
a  higher  overall  percentage  of  remarks  in  these  categories  than  Experts 
(Novice  Action  Assessments  =  42. 8X,  Expert  Action  Assessments  =  32. 6%).  The 
direction  of  this  difference  is  supported  by  the  Replication  criterion.  This 
is  further  support  for  the  hypothesized  differences  in  the  deliberation 
strategies  of  Experts  and  Novices.  Of  particular  interest  is  the  Deliberation 
category  in  probe  point  1.1  which  meets  the  Large  Difference  criterion.  This 
category  provides  the  most  direct  support  for  the  hypothesis,  generated  on  the 
basis  of  retrospective  interviews  with  FGCs  (Calderwood  et  al . ,  1987),  that 
Novices  deliberate  about  options  more  than  do  Experts.  The  difference  found 
in  the  Contingency  category  of  probe  point  1.5  also  favors  the  Novices. 
Contingency  planning  is  conceptually  related  to  the  Deliberation  code.  In  one 
case  the  deliberation  is  about  what  to  do  now,  in  the  other  it  is  about  what 
to  do  in  the  future. 

The  fact  that  Experts  have  a  higher  percentage  of  Cue-deliberation 
remarks  than  Novices,  while  the  opposite  is  true  for  Action-deliberation 
remarks  should  not  be  interpreted  to  mean  that  Novices  make  absolutely  more 
Action-deliberation  remarks  than  Cue-deliberation  remarks.  In  probe  points 
1.1  and  2.1,  both  Experts  and  Novices  deliberate  more  about  Situational  cues 
than  about  action  alternatives.  These  relationships  are  portrayed  in  Figure 
2.  This  is  an  interesting  finding,  given  that  most  decision  models  are  mute 
on  the  nature  of  problem  structuring  and  classification. 


1-30 


H  Cue-deliberation 
□  Ac tl on-del i bera  ti on 


n-i 


Probe  1.1  Probe  1.5  Probe  2.1 

£  *  Expert 
N  «  Novice 


Figure  2a  Percent  of  Cue-deliberation  and  Action-deliberation  for 
Experts  and  Novices  in  Selected  Probe  Points 


Over  the  three  probe  points,  there  were  slightly  more  Goal  remarks  for 
the  Experts  than  for  Novices  ( Expert  Goals  =  6.6X,  Novice  Goals  =  5.5X),  and 
the  relationship  meets  the  Replication  criteria.  This  is  not  a  surprising 
finding  given  the  general  assumption  that  intelligent  performance  is 
distinguished  by  being  "goal -driven"  (e.g.,  Larkin,  1981;  Holding,  1985; 
Anderson,  1981).  What  did  surprise  us  was  how  uninformative  the  goal 
statements  usually  were  for  illuminating  the  basis  for  action.  Indeed,  Action 
remarks  were  frequently  hard  to  discriminate  from  Goal  remarks.  Take,  for 
example ,  the  remark  "I  would  order  a  line  inside  to  locate  and  attack  the 
fire."  This  could  be  parsed  as  Action  =  take  line  inside;  Goal  =  locate  fire 
and  Goal  =  attack  fire.  Another  remark,  "I  would  order  the  engine  crew  to 
attack  the  fire."  The  goal  "to  attack  the  fire"  is  stated  as  an  action  but 
really  means  the  same  thing  as  the  previous  remark.  Some  support  for  the 
confuseability  of  actions  and  goals  comes  from  the  reliability  assessments  of 
these  categories.  Considered  separately,  the  reliability  of  the  categories 
for  both  sets  of  coders  was  in  the  "good"  range  (kappa  <.75).  Collapsing  the 
categories  together  raises  the  kappa  into  the  excellent  range  (kappa  >.75)  for 
both  sets  of  coders. 

The  fact  that  some  of  the  differences  are  present  in  some  probe  points 
and  not  others  should  not  surprise  us.  Each  situation  would  be  expected  to 
create  its  own  unique  context  that  will  highlight  specific  components  of  a 
decision  process.  We  are  far  from  being  able  to  specify  the  conditions  under 
which  the  observed  results  will  occur.  In  the  next  section  we  describe  our 
analysis  of  the  content  of  these  protocol  categories  as  a  beginning  in 
understanding  these  conditions. 
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We  assume  that  expertise  represents  a  kind  of  operative  knowledge  that 
allows  experts  (in  this  case  decision  makers)  to  perform  under  a  wide  range 
of  conditions  in  their  domain.  It  is  a  capacity  to  achieve  problem  solutions, 
and  not  a  property  of  behavior  (see  for  example  Johnson,  Zu&lkeraan,  &  Garber, 
1987).  Thus,  the  "correctness"  of  a  decision  is  not  a  sufficient  index  of 
proficiency.  Not  only  can  right  decisions  be  made  for  the  wrong  reasons  and 
vice  versa,  but  in  real-world  tasks  it  may  be  impossible  to  define  what  a 
"correct"  decision  is  (Edwards,  Kiss,  Majone,  &  Toda,  1984).  The  goal  of 
understanding  how  experts  structure  and  represent  knowledge  is  impetus  for  the 
rise  in  studies  based  on  content  analysis  of  protocols  in  recent  years  (e.g., 
Graesser,  1981;  Hammond,  Hamn,  Grassia,  Sl  Pearson,  1984). 

For  this  study,  we  have  developed  our  own  method  of  representing  the 
content  of  the  protocol  remarks.  The  method  was  designed  to  illuminate 
commonalities  in  a  way  that  would  aid  interpretation  of  Expert/Novice 
differences  in  relation  to  the  RPD  model.  The  method  is  based  on  node  units 
which  describe  the  content  or  topics  of  the  remark  categories.  A  node  is 
defined  as  being  present  whenever  remarks  in  at  least  two  protocols  are  judged 
liave  referred  to  the  same  topic.  The  assumption  is  that  for  two  individuals 
to  comment  on  the  same  cues,  make  the  same  inference,  or  require  the  same 
action,  the  remark  reflects  some  component  of  the  underlying  schemas  that  have 
been  activated  by  the  simulation  task.  Nodes  are  indicated  by  a  word  or 
phrase  summarizing  the  remark  topic. 

Hie  first  step  was  to  examine  the  content  of  the  coded  remarks  and  to 
organize  them  into  conceptually  meaningful  sets.  This  was  done  by  listing  the 
content  of  all  remarks  in  a  given  category  for  a  set  of  protocols.  Each 
protocol  was  then  analyzed  for  whether  it  contained  an  instance  of  each  remark 
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item.  If  two  or  more  protocols  had  the  same  item,  it  became  a  node  to  be 
represented  on  the  node  graph  for  that  probe  point. 

A  decision  had  to  be  made  about  how  best  to  display  nodes  graphically. 

One  possibility  was  to  organize  the  nodes  into  a  temporal  ordering  that  might 
reflect  causal  connections  between  cue  nodes  and  action  nodes.  However,  the 
temporal  ordering  of  categories  varied  considerably  and  no  mechanism  was  found 
for  abstracting  the  temporal  relationships.  In  the  absence  of  theory  that 
would  generate  a  logical  or  natural  ordering  of  the  nodes,  we  decided  to 
retain  a  simple  list  structure  that  would  indicate  the  degree  of  overlap  in 
the  sample  of  protocols  for  the  nodes.  In  other  words,  we  wished  to 
distinguish  nodes  which  were  mentioned  in  all  or  many  of  the  protocols  from 
those  for  which  there  was  minimal  overlap.  This  index  may  reflect  the 
typicality  or  centrality  of  the  node  for  the  underlying  knowledge. 

Kour  classes  of  nodes  were  defined: 

1 )  Cue  Nodes  =  Cues  and  Cue-elaborations 

2)  Knowledge  Nodes  =  General  Knowledge,  SOPs,  Cue-deliberation,  and 
Anticipation 

3)  Action  Nodes  =  Actions,  Action-elaborations,  and  Action- 
deliberation 

4)  Goal  Nodes  =  Goals  and  Goal-elaborations 

The  content-node  analysis  was  carried  out  on  the  two  initial  probe  points 
(probe  point  1.1  and  probe  point  2.1).  The  other  probe  point  (probe  point 
1.5)  represented  a  very  novel  situation  for  the  FGCs  we  studied  (another  fire 
alarm  was  received  while  they  were  still  engaged  in  fighting  the  apartment 
fire)  and  it  did  not  seem  to  have  sufficient  overlap  in  the  action  items  to  be 
meaningful.  The  overlap  in  the  situation  assessment  categories  was  also  quite 
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slim,  perhaps  because  it  was  a  "later"  decision  and  some  important  information 
was  not  repeated.  Additional  work  is  needed  before  we  will  be  able  to  say  how 
general  the  proposed  method  is. 

In  the  node  graphs  (Figures  3-6),  symbols  are  used  to  distinguish  each  of 
the  node  categories.  The  open  symbols  indicate  that  the  node  type  was  present 
in  fewer  than  5  of  the  protocols  within  the  group  (low  density);  hatched 
symbols  indicate  overlap  on  five  to  seven  of  the  protocols  (medium  density); 
and  filled  symbols  indicate  overlap  on  eight  to  eleven  of  the  protocols  (high 
density).  The  side-by-side  presentation  of  the  Expert  and  Novice  graphs 
facilitates  noticing  which  nodes  are  absent  for  either  group  (indicated  by  a 
"?")  and  differences  in  the  node  densities.  In  the  Cue  graph,  some  of  the 
nodes  seem  to  fall  into  natural  groupings  that  are  also  indicated.  The  label 
for  the  grouping  is  bracketed  to  indicate  that  it  is  based  on  our  own  grouping 
strategy  rather  tiian  being  tied  directly  to  any  features  of  the  protocols 
themselves. 

Many  of  the  differences  between  Experts  and  Novices  in  the  node  graphs 
are  based  on  very  small  samples,  s>  care  must  be  eaken  not  to  overemphasize 
any  particular  difference.  What  we  are  seeking  are  general  patterns  of 
contrast  that  can  be  used  to  generate  hypotheses  for  future  research  or  that 
can  be  meaningfully  related  to  specific  contextual  variables. 

Occasionally  the  node  analysis  revealed  a  "branch"  in  the  nodes 
representing  a  two-choice  alternative,  usually  of  the  form  "X  or  not-X,"  so 
these  nodes  are  shown  linked  together  on  the  graph.  Linked  nodes  are 
particularly  interesting  because  they  seem  to  reflect  probabilistic  inferences 
or  option-selection  processes  that  would  not  have  been  apparent  in  a  single 
protocol.  That  is,  a  remark  topic  may  indicate  only  the  outcome  of  a  decision 
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process  in  a  single  protocol.  However,  by  seeing  the  cijuiplrmonl-nry  outcome 
t.hat  is  reached  by  other  individuals,  one  can  make  inferences  about  the 
underlying  decision  processes. 

Probe  Point  1 . 1 

Interpretation  of  the  nodes  for  probe  point  1.1,  reqiores  knowing  the 
context  for  the  decision  event.  In  this  scenario  the  participant  PGC  hears  an 
alarm  to  an  apartment  fire.  After  receiving  details  about  the  time  of  day, 
weather  conditions,  and  responding  units,  he  learns  several  facts  about  the 
structure  that  were  known  to  the  FGC  of  the  incident  —  that  this  is  a  poorly 
constructed  building,  that  it  has  punk-board  flooring  and  a  second  roof  added 
to  the  original  .  He  also  hears  a  descript  ion  of  the  entrances  fuid  riumlxT  of 
apartments  in  the  building.  As  he  "travels"  to  the  scene,  he  hears  a  size-up 
being  disjwtched  from  the  first-arriving  officer  on  the  scene.  He  also  hears 
the  order-  for  the  engine  crew  to  take  an  inch-and-a-l»aJ  f  line  into  the 
basement.  Upon  "arrival"  the  participant  is  shown  a  slide  depicting  the  scene 
as  it  would  appear.  The  scene  shows  details  of  the  apartment  structure,  heavy 
smoke  is  shown  covering  the  building  front,  and  flames  are  shown  escaping  from 
a  basement,  apartment.  TTie  first-arriving  crews  are  shown  exiting  the 
basement.  The  participant  FGC  then  hears  a  report  from  the  first-arriving 
officer  indicating  that  the  intense  smoke  and  heal  arc  preventing  crews  from 
reaching  the  fire. 

The  Cue  and  Knowledge  node  graph  for  this  probe  point  is  sho’r>  in  Figure 
3.  This  graph  represents  a  simplification  of  49  separate  Cu^  remark  topic*1 
and  27  separate  Knowledge  topics. 

'Hie  Cue  nodeB  fall  into  categories  of  fire  dynamics,  the  structural 
features  of  the  building,  the  building's  occupancy,  personnel  and  equipment 
resources,  and  current  tactics.  Although  these  node  categories  are  present 
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Figure  3:  Cue  and  Knowledge  Nodes  fop  Probe  Point  1.1 


for  hol.h  the  Exi»or1.s  and  Novices,  there  ure  some  specific  differences  in  the 
nodes  t.hat  are  worth  considering.  'ftie  FL\i«ert  node  labeled  "look  of  the  fire" 
refers  to  the  iriterisi  ty  and  seoj>o  of  flu*  fire  suggested  in  the  grajdiic 
illustration.  There  is  no  corresponding  node  for  the  Novices,  who  may  be  less 
able  to  notice  the  subtleties  of  the  perceptual  cues.  The  unspecified  content 
of  "look"  is  consistent  with  the  idea  that  perceptual  cues  are  among  the  most 
difficult  to  articulate.  The  Critical  Decision  method  used  in  other  studies 
in  this  series  was  designed  to  probe  for  clarification  of  the  nature  of  such 
cues. 

The  absence  of  the  structure  nodes  ("poor  construction"  and  "punkboard 
flooring")  in  the  Expert  graph  is  interesting.  These  cues  were  reported  in 
the  audio  portion  of  the  scenario,  but  are  not  directly  present,  in  the  graphic 
illustration  of  the  scene.  The  analysis  cannot  tell  us  whether  the  presence 
of  the  nodes  mefins  t.he  cue  is  more  import-mt  to  the  Novices  or  whether  it  is 
simply  unstated  by  the  Experts  for  some  reason.  The  Novices  may  need  to 
remind  themselves  of  previously  stated  cues  more  than  Experts. 

'Jhe  Experts  made  more  remarks  pertinent,  to  the  availability  of  personnel 
and  equipment,  in  line  with  the  idea  tliat  they  more  easily  adopt  a  "command" 
perspective  that  involves  managing  these  resources.  At  a  more  global  level, 
the  Experts’  graph  has  more  high-density  Cue  nodes  tlian  the  Novices,  possibly 
indicating  a  more  coherent  schema  underlying  these  topic  statements. 
Alternatively,  density  differences  could  indicate  more  or  less  overall  remark 
topics,  but  this  does  not  seem  to  be  the  case  here.  Experts  and  Novices  had 
roughly  equal  numbers  of  topics  represented  In  the  protocols  (35  and  31, 
respectively) . 
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Differences  in  the  Knowledge  nodes  are  among  the  most  interesting  l>ecHuse 
those  arc-  closest  to  the  inferential  and  reasoning  processes  that,  might 
distinguish  different  decision  outcomes.  The  nodes  were  extracted  from  a 
total  of  27  topics  in  the  Knowledge  categories  with  an  identical  number  of 
unique  topics  (20)  for  the  Expert  and  Novice  protocols.  The  most  striking 
difference  in  these  Knowledge  nodes  is  the  fact  that  a  branch  present  in  the 
Expert  graph  is  missing  in  the  Novice  graph.  This  branch  represents  a 
dichotomy  between  a  judgment  that  the  presently  available  resources  are  or  are 
not  adequate.  Recall  that  the  high  density  Cue  node  indicating  attention  to 
resource  availability  was  also  absent  in  the  Novice  graph.  It  seems  safe  to 
assume  that  issues  of  resource  allocation  were  more  salient  to  the  Experts. 

Another  node  present  in  the  Expert  liut  not  the  Novice  gruiJi  concerns  the 
topic  of  "focus."  Some  of  the  Expert,  protocols  contained  a  reminder  to  pay 
attention  to  the  whole  situation  and  not  just  the  involved  apartment.  This 
node  lias  the  flavor  of  a  maxim  or  general  rule.  The  idea  of  learning  to 
expand  one’s  focus  beyond  the  most  salient  or  immediate  problem  lias  been  made 
frequently  in  our  interviews  with  firefighters.  The  single  Knowledge  node 
that  is  present  in  the  Novice  graph  but  not  in  the  Expert  graph,  "life  is  top 
priority,"  also  has  this  maxim  or  rule-like  quality. 

Turning  to  the  Action  and  Goal  node  graph  (Figure  4)  one  can  agAin  find 
several  differences  between  the  Ex^xm-Ls  and  Novices.  The  Action  nodes 
represent  the  overlap  from  a  total  of  35  distinct  remark  topics,  32  for  the 
Experts  and  22  for  the  Novices.  For  the  first  three  nodes,  representing  the 
most  overlap  in  the  remark  topics,  the  same  nodes  are  present  in  both  graphs, 
but  there  was  somewhat  more  overlap  among  the  Expert  protocols.  Although  both 
graphs  contain  the  "back-up  line  to  basement"  node,  there  is  a 
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( ;»g  on  t.he  Novices’  node  indicating  tho  possibility  of  taking  the  lino  in  the 
roar  on  trance.  It  w/ts  cleurly  stated  in  the  scenario  Dial  there  was  no  rear 
entrance,  so  this  acLiori  indicates  an  error. 

The  Expert  Knowledge  graph  contains  a  branch  for  whether  to  implement  a 
search  and  rescue.  The  issue  here  is  whether  to  accept  the  "all  clear"  cue 
given  in  tire  scenario.  Although  more  of  the  Experts  indicated  that  the  search 
and  rescue  would  he  needed  (see  the  high -dens  i  t.y  node),  it  was  clearly  more  of 
an  issue  for  these  officers  then  for  the  Novices.  Only  half  of  this  branch  is 
represented  in  the  Novice  graph,  and  only  by  a  low-density  node.  Four  of  the 
low-density  nodes  present  in  the  Expert  graph  are  absent  in  the  Novice  graph. 
All  of  these  represent  potentially  im|iortant.  tnel.icnl  considerations,  whereas 
the  single  node  which  is  absent  from  the  Expert  grajh  represents  a  tactic  that 
has  already  been  accomplished  in  the  scenario. 

There  are  three  Goal  nodes  in  the  Expert  grajh  .u«l  only  one  in  the 
Novice,  but  these  are  all  low-density  nodes  representing  straight-forward 
outcomes  of  the  specified  actions.  These  are  general  goals  that,  like 
standard  operating  procedures,  represent  fire  ground  tactics  that  would  apply 
to  almost  any  structural  fire  of  this  kind.  Sitiiation-specific  goals  are 
curiously  absent. 

Probe  Point-  2.1 

In  this  scenario,  a  report  is  received  of  a  fire  at  a  restaurant  housed 
in  a  well-known  historical  building.  The  narrator  supplies  information  about 
the  time  of  day,  the  weather,  and  the  building  construction.  The  building  is 
said  to  be  brick  and  heavy  timber,  to  have  been  renovated  and  enlarged  with 
the  additions  being  balloon  construction.  The  responding  units  are  indicated 
and  the  size-up  is  heard  over  the  radio  during  the  porticipant-FGCs  transit  to 
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Ihe  scene,  TTie  pai  ticii*uit-FGC  is  shown  a  slide  indicating  the  details  of  the 
scene  as  it  would  appear  on  his  arrival  and  indicating  the  tactical  maneuvers 
that  were  taken  by  the  first-arriving  officer. 

The  Cue  and  Knowledge  nodes  for  this  probe  point,  are  shown  in  Figure  5. 
Roughly  equivalent  Cue  categories  are  present  in  the  Expert  and  Novice  graphs, 
although  there  are  several  minor  differences  in  the  specific  node  topics. 

More  striking  are  the  differences  in  the  Knowledge  nodes.  The  high-density 
"water  supply  problem"  node  in  the  Expert,  graph  is  not  even  present  in  the 
Novice  graph,  showing  a  lack  of  attention  to  resource  issues  that  was  also 
seen  in  probe  point  1.1.  A  decision  branch  related  to  the  probabilit>  that, 
the  fire  started  in  the  upper  floors  versus  the  basement  shows  up  in  the  two 
medium-density  nodes  indicated.  Neither  of  these  nodes  is  present  in  the 
Novice  graph,  nor  is  the  node  indicating  that  ventilation  may  be  difficult. 

Ihe  only  Knowledge  node  present  in  the  Novice  graph  but.  not  in  the  Expert 
graph  was  an  appraisal  indicating  that  the  outcome  of  the  incident  would 
probably  not  be  successful . 

The  Action  and  Goal  node  graphs  for  the  Experts  and  Novices  (Figure  6) 
are  not  as  strikingly  dissimilar  as  they  were  for  probe  point  1.1.  Rather 
than  the  richer  and  more  elaborated  actions  for  the  Experts  for  that  probe 
point,  here  the  numbers  of  distinct,  action  topics  for-  the  two  groups  were 
almost  identical  (24  and  23  for  Experts  and  Novices,  respectively).  The  most 
notable  difference  is  perhaps  the  presence  of  a  branch  in  the  Expert  graph 
between  the  "ventilate"  and  "wait  to  ventilate"  nodes.  Knowing  when  to 
ventilate  is  one  of  the  most  frequently  mentioned  characteristics  of  expertise 
in  this  domain.  The  low-density  branch  in  the  Novice  graph  for  the  nodes 
"take  second  line  in  front"  and  "take  second  line  in  rear"  that,  is 
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Action  and  Goal  Nodes  for  Probe  Point  2.1 


not  present  in  the  Exi**rt  graph  is  hard  Lo  interpret,  given  that  the  second 
engine  has  not  yet  arrived.  Exports  may  simply  have  been  better  at  limiting 
their  remarks  to  actions  that  would  be  taken  at  the  time  of  the  probe.  This 
possibility  can  be  examined  when  probe  point  2.2  is  analyzed  in  the  near 
f  uture . 

No  remarkable  differences  are  evident  in  the  Goal  nodes.  Again,  the 
goals  tend  to  be  the  general  goals  of  good  fire  ground  tactics  and  do  not  seem 
to  illuminate  the  present  situation  or  factors  associated  with  expertise. 
Evidence  for  Decision  Strategies 

The  analysis  of  the  remark  topics  presented  thus  far  does  not  directly 
address  the  evaluation  strategies  described  by  the  RPD  model.  Such  strategies 
might  only  be  discerned  by  considering  the  meanings  of  a  series  of  remarks 
taken  together,  and  in  context.  Possible  clues  to  these  strategies  were  noted 
on  each  protocol  and  examined  separately  from  the  remark  frequency  and  content 
analyses.  In  addition,  we  tried  to  be  sensitive  to  instances  of  analogue  use, 
prototypes,  and  errors  of  judgment,  or  interpretation. 

The  data  relevant  to  these  processes  were  disappointingly  sparse. 

Tn  the  66  protocols  examined  and  coded  for  this  st.udy,  we  found  only  11 
instances  of  deliberated  decisions,  and  a  handful  of  instances  of  progressive 
deepening,  specific  analogues,  prototypes  and  imagery. 

Nonetheless,  the  examples  that  were  found  offer  a  tantalizing  look  at  the 
cognitive  processes  and  strategies  that  underlie  command  decision  making.  Of 
the  11  cases  of  deliberated  decision  making  identified  in  the  protocols,  six 
were  cases  of  serial  decision  strategies  —  two  involving  cue-deliberation  and 
four  involving  action-deliberation.  In  the  remaining  five  cases  of  concurrent 
decision  strategies,  four  involved  Cue-deliberation  and  one  involved  Action- 
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deliberation.  Excerpts  from  (lie  protocols  containing  these  de] iterated 
decisions  are  presented  in  Tables  6  and  7. 

leased  on  these  few  instances,  we  offer  several  highly  tentative 
observations : 

First,  the  instances  of  deliberated  decision  making  do  not  appear  to  be 
accounted  for  by  the  simulated  incidents  themselves.  They  do  not  consistently 
occur  at  any  particular  point  in  the  incident  or  in  response  to  particular 
aspects  of  the  situation  represented. 

Second,  there  does  not  appear  to  be  conmonality  in  what  the  FGCs  are 
deliberating  about  (e.g.,  apparatus  placement,  where  the  fire  is  located, 
resource  availability).  There  is  virtually  no  overlap  in  the  content  of  these 
deliberated  decisions. 

Third,  examination  of  the  serially  deliberated  decisions  suggest  that 
this  strategy  is  activated  when  the  PGC  notes  new  or  previously  unnoticed  cue 
information.  Even  when  the  serial  strategy  clearly  involves  action 
deliberation,  it  does  not  appear  to  occur  because  of  some  previously 
unrealized  action  possibility. 

liecognizing  that  the  absolute  frequencies  of  occurrence  are  very  low,  we 
would  note  nonetheless  that  comparison  of  Experts’  and  Novices’  data  reveals 
directional  differences  in  line  with  other  Expert/Novice  findings  obtained  in 
this  program  of  research.  Of  the  11  deliberated  decisions,  64%  were  obtained 
from  Novice  PGCs’  protocols.  Moreover,  Novice  FGCs’  deliberated  decisions 
more  often  involved  concurrent  than  serial  strategies  (57%  vs.  43%).  This  was 
not  the  case  for  the  Expert  FGCs,  whose  deliberated  strategies  more  often 
involved  serial  than  concurrent  strategies 
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TABLE  6 

of  SA  Deliberation 


i-u 


Instances  of  Action  Deliberation 


(75%  vs.  25%).  This  does  offer  support  for  the  notion  that  Novices  axe  more 
likely  tJian  Experts  to  employ  deli tierat.ion  in  decision  making. 

Summary  and  Conclusions 

The  present  investigation  was  carried  out  as  the  final  study  in  a  series 
of  interrelated  studies  of  command-and-conlrol  decision  making.  As  such,  it 
has  been  guided  in  conceptual i ration  and  in  choice  of  methods  by  the  rich  and 
often  provocative  data  obtained  in  those  previous  efforts.  The  results  of  our 
earlier  studies  —  carried  out  in  a  variety  of  natural  settings  —  had  raised 
a  number  of  questions  about  the  validity  and  utility  of  standard  decision 
models  for  unde  {landing  dec  ision  tx'lmviors  in  the  f in*— pressured ,  high  risk 
and  complex  situations  represented  by  cormand-and -control .  These  studies 
served  as  the  hnsis  fox  develoixnent  of  our  RPl)  node]  ,  which  has  continued  to 
guide  our  thinking,  and  which  we  see  as  offering  an  alternative 
conceptual izat. ion  to  standard  decision  models  that  emphasize  option  generation 
and  evaluation. 

As  results  from  these  several  studies  accumulated,  and  as  we  developed 
and  refined  our  Critical  Decision  method,  it  became  clear  that  we  needed  to 
examined  certain  key  issues  under  more  controlled  conditions.  For  example, 
tlie  studies  offered  repeated  and  compelling  suggestions  of  the  importance  of 
situation  assessment,  processes  for  decision  making,  especially  as  it  is 
carried  out  by  highly  proficient  decision  makers.  But  wildiout  the  me/ins  to 
know  more  precisely  what  information  was  available  in  a  situation,  and  to 
present  the  same  set  of  situation  features  and  pivotal  events  within  «  given 
incident,  we  were  left  at  an  uncomfortubly  speculative  level  in  terms  of  model 
testing  and  development.. 
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The  simulations  developed  for  this  study  were  designed  to  reproduce  as 
closely  as  possible  the  experience  of  fire  ground  command  during  an  actual 
incident.  The  intensity  of  involvement  we  observed  in  our-  parti ciparits,  as 
well  as  the  sheer  amount  of  talk  they  generated  in  response  to  the 
simulations,  indicates  that  we  were  able  to  represent  the  key  elements  of  an 
actual  incident  with  enough  authenticity  to  engage  these  FGCs’  knowledge  and 
decision  processes.  The  '‘think-aloud’’  method  also  offers  evidence  of  the 
cognitive  content,  and  processes  that  underlie  decision  making,  without  the 
potential  response  biases  introduced  by  the  guided  probes  and  semi -structured 
interview  methods  of  CDM. 

The  remark  frequencies  arc  assumed  to  provide  an  index  of  the  relative 
attention  given  to  different  aspects  of  a  decision  event.  These  data 
substantiated  the  critical  role  that  situation  assessment,  in  command  decision 
making.  In  addition,  convergent  evidence  was  provided  for  the  hypothesized 
relation  between  these  processes  and  relative  degrees  of  domain  skill. 

i  or  Expert  PGCs,  remarks  related  to  situation  assessment  consistently 
exceeded  those  given  to  action  assessment,  while  the  opposite  was  true  for 
Novices. 

The  content  node  analysis  of  the  protocol  remarks  was  undertaken  in  order 
to  investigate  the  nature  of  the  schema  that  are  presumed  to  underlie  the 
commanders’  decisions.  The  method  allowed  several  intererting  features  of  the 
protocols  to  bo  illuminated.  The  node  graphs  for  the  Experts  tended  to  be 
richer  and  more  elaborated  than  the  Novices’,  but  the  qualitative  differences 
in  specific  nodes  were  the  most  interesting.  The  Experts’  graphs  revealed 
different  issues  being  addressed  on  the  basis  of  a  similar  set  of  cues.  These 
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II 


issues  involved  the  very  types  of  causal  inferences  that,  one  would  expect  to 
be  associated  with  more  highly  developed  domain  knowledge. 

One  unexpected  finding  that  emerged  from  the  node-graplang  procedure  was 
the  ability  to  detect  node  branches  that  were  not  readily  discernible  in  the 
individual  protocols.  In  these  cases,  it  is  assumed  that,  a  split  in  the 
"solutions"  reached  about  some  ambiguity  in  the  situation  reflect  an 
underlying  decision  process  that  was  only  infrequently  articulated  in  the 
protocols.  If  this  assumption  is  correct  then  we  have  demonstrated  that  the 
decision  "space"  is  not  the  same  for  individuals  at  different  levels  of  skill 
and  experience.  This  is  quite  different  from  the  standard  view  which  treats 
dif Terences  in  decision  making  in  terms  of  assigning  probabilities  and  values 
to  a  set.  of  pre-defined  alternatives.  We  hope  to  pursue  the  implications  of 
this  idea  in  future  research. 

We  were  also  surprised  at  the  relatively  low  frequencies  of  goal  remarks, 
by  either  Expert  or  Novice  FGCs.  Moreover,  it  seemed  to  us  tliat  when  goals 
were  discussed,  there  was  often  a  nebulous,  generic  quality  to  them  that  was 
quite  different  from  the  situation  and  action  assessments  being  offered.  We 
have  been  struck  by  the  consistent  difficulty  we  have  had  across  this  series 
of  studies  in  getting  people  to  talk  informatively  about  goals.  It  has  begun 
to  occur  to  us  that  the  problem  may  not  be  one  of  inadequate  research  methods. 
Rather,  people  may  have  an  extremely  difficult  time  thinking/talking  about 
goals  independently  of  the  actions  they  supposedly  guide  or  the  situations 
they  are  intended  to  address.  Given  the  many  decision  support  systems  that 
are  organized  uround  goal  specification  and  clarification,  we  had  expected  to 
find  evidence  of  the  utility  of  goals  in  naturally  occurring  decision  making. 
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We  think  the  failure  to  find  such  evidence  in  this  study  or  in  others  in  this 
series  is  intriguing,  and  plan  explore  this  aspect  of  decision  making  further. 

Finally,  findings  from  the  present,  study  indicate  that  evidence  on 
cognitive  processes  and  decision  strategies  is  not  often  revealed  in  "think 
nloud"  protocol  data.  The  Critical  Decision  method  was  originally  developed 
to  elicit  such  information  in  the  context  of  retrospective  reports  of  actual 
events.  Pairing  the  simulation  format  with  CDM  would  appear  to  offer  a 
powerful  research  tool  for  studying  this  aspect  of  decision  making  under  more 
controlled  conditions.  An  initial  study  using  this  approach  is  presently 
under  way  as  part  of  another  contract  (MDA903-86-C-0170)  and  the  results  look 
promising.  Nonetheless,  when  FGCs’  protocols  did  contain  evidence  of 
cognitive  process,  it  was  clear-cut  and  compelling.  The  protocols  provide 
supportive  evidence  for  the  RPD  model.  They  indicate  that  people  do  use 
serial  decision  strategies,  arid  that  Novices  rather  than  Experts  are  likely  to 
deliberate  during  decision  making. 

At  a  more  general  level,  the  protocols  have  led  us  to  reconsider'  certain 
aspects  of  the  RPD  model.  The  supposition  that,  decision  making  occurs  as  the 
outcome  of  a  production  rule:  if  CONDITION,  then  do  ACTION  suggests  a 
linearity  to  decision  events.  That  is,  the  decisis  maker  assesses  a 
situation,  recognizes  i  t  as  familiar,  and  proceeds  to  act  —  guided  in  his  or 
her  choice  of  what  to  do  by  that  sense  of  familiarity.  We  are  increasingly 
less  comfortable  with  the  ordered  quality  of  thit-  conceptualization,  while 
continuing  to  adhere  to  it.s  recognitional  components. 

We  would  acknowledge  that  division  of  decision  making  into  separate, 
independent  situation  assessment  and  action  components  is  u  useful  convention, 
especially  for  comparing  the  RPD  model  with  other  models  of  decision  making. 
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Nonetheless i  we  are  increasingly  convinced  that  the  division  is  an  artificial 
one.  We  did  not  find  FGCs  talking  first  about  the  situation  and  once  their 
concerns  about  the  nature  of  the  situation  and  been  satisfied,  only  then 
moving  on  to  action  remarks.  Rather,  their  attention  seemed  to  move  back  and 
forth  between  elements  of  the  situation  and  the  actions  intended  to  address 
them.  They  are  constantly  assessing  BOTO  situational  factors  and  action 
factors  —  recognizing  categories  of  each  and  matching  one  to  the  other  as 
needed. 
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APPENDIX  A 


PROBE  POINTS 

EXPERT  FGC  -  PROTOCOL  *06 
Probe  Point  1.1 


Okay,  they  tell  me  everyone  is  out.  I  am  not  sure  if  that  meant  out  of 
that  apartment  or  out  of  the  building.  I  am  going  to  assume  that  there  could 
still  be  someone  iri  the  building  itself,  I  am  concerned  with  the  whole 
building,  not  with  just  the  apartment. 

I  want  to  ventilate  the  building  so  that  those  crews  can  get  into  that 
j "articular  apartment.  I  will  probably  ladder  the  front  of  the  building;  I 
will  have  crews  at  the  rear  of  the  building  with  ladders;  I  want  a  crew  to  go 
in  and  check  the  rest  of  the  building  at  this  end  and  search  and  rescue  if 
needed  down  in  that  areu. 

If  the  ventilation  can  help,  and  they  get  the  fire,  that  is  fine.  In  the 
meantime,  I  am  going  to  call  for  some  additional  medics,  I  only  had  one  sent, 
in  a  fire  like  this  I  want  more  on  the  scene.  If  I  don’t  need  them  I  can 
release  them  quickly  enough.  I  will  probably  stage  some  extra  apparatus  until 
1  am  sure  we  can  contain  this  fire  in  this  apartment.  If  it  doesn’t  appear 
that  the  ventilation  of  the  fire  is  going  successfully,  then  I  can  use  that 
equipment,  it  will  be  on  hand. 

I  will  probably  send  a  crew  to  the  roof,  just  to  stand  by;  but  I  don’t 
want  them  doing  anything  just  yet.  I  also  will  have  some  extra  lines,  they 
have  an  inch  and  three-quarter,  I  want  another  line  at  least  up  above  the  fire 
and  also  between  the  fire  and  the  uriexposed  portion  of  the  building.  That’s 
all. 
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EXPERT  FGC  -  PROTOCOL  #09 


Probe  Point  1 . 1 


Okay,  they  said  everyone  was  out,  by  that  I  would  be  assuming  that  they 
mean  this  whole  building  had  been  cleared. 

The  fire  is  in  the  basement.  It  is  hard  to  tell,  flames  coming  from  just 
this  one  window  in  this  one  area.  Probably  would  call  for  extra  equipment, 
get  a  second  alarm  there,  depending  on  the — I  don’t  know  if  I  would  call  a 
second  alarm  with  just  that  amount  of  fire  showing.  Get  another  line  down  in 
the  basement,  try  to  get  it  ventilated  so  crews  could  get  back  in  there  and 
also  try  to  get  crews  up  on  the  other  floors  to  conduct  a  search  to  be  sure 
everyone  is  out.  The  two  minutes  that  the  crews  have  been  on  the  scene,  it  is 
unlikely  that  they  would  have  been  able  to  complete  a  real  search  of  a 
Ituilding,  so  second  in  crew  probably  to  ventilate  in  tliat  area  of  the  fire. 

Get  another  backup  line,  try  and  drive,  with  two  lines  together,  try  to  get 
down  to  the  fire  area.  When  other  people  got  there,  liave  a  third  line  taken 
to  the  second  floor,  the  floor-  above,  check  for  fire  extension  vertically. 

With  the  poor  construction,  if  it  has  been  burning  long  enough  to  be  popping 
out  the  window  like  tliat,  it  is  quite  likely  that  it  has  been  spreading 
upward.  I  would  like  to  have  at  least  two  lines  in  the  basement,  one  line  on 
the  second  floor  and  people  searching  the  second  and  third  floor.  Old 
buildings  are  prone  to  have  open  vertical  shafts,  fire  can  spread.  If  that  is 
the  case,  then  wait  for  initial  reports  from  the  other  crews  searching  the 
building.  If  tliat  would  be  the  case,  I  would  probably  complete  the  second 
alarm. 

Still,  looks  basically  like  an  offensive  fire,  just  need  to  clear  it  out 
a  little  bit  to  have  people  make  the  attack.  Looks  pretty  simple  right  now. 
Again,  I  can’t  tell  from  the  picture  how  the  stairs  to  the  upper  floors  would 
run.  Are  there  separate  stairs  for  euch  set  of  apartments  there?  They  might 
have  said,  but  I  missed  it.  Protect  the  stairs,  make  sure  they  stay  clear, 
make  sure  we  got  everybody  out. 

Ventilation  to  get  people  to  the  fire  to  put  the  fire  out,  eliminate  the 
hazard  and  also  to  remove  the  people  from  the  hazardous  area.  Pretty  much 
simultaneous  operations.  We  will  just  assign  the  crews  and  wait  for  further 
reports . 
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NOVICE  PGC  -  PUfTTOOOl,  #02 
Probe  Point.  1 . 1 


Oh,  gosh,  picturing  the  whole  fire  scene,  picturing  the  size  of  the 
building  which  creates  a  problem.  Immediately  I  thought  tli&t  with  that  large 
of  a  building,  I  would  have  called  additional  equipment  and  they  said  it  was 
occupied.  Poor  construction  is  another,  being  aware  of  that,  I  would  have 
another  reason  for  calling  additional  equipment. 

I  do  not  think — I  wouldn’t  have  permitted  police  crews  inside  the 
building,  for  just,  that,  reason.  They  have  no  eejui intent.  They  would  be 
exposed  to  who  knows  what.  I  would  have  probably  been  setting  up  the  other 
engine  while  they  were  in  route.  I  would  liave  had  them  sj>ot  certain  areas, 
supplied  the  actual  first  apparatus  on  the  scene.  The  ladder  truck,  there 
again  like  I  said,  I  would  have  probably  called  for  additional  equipment  being 
it  was  occupied.  Would  not  have  permitted  the  police  in  there,  T  would  have 
hud  my  personnel  evacuate  the  building,  using  their  protective  equipment  and 
breathing  apparatus  which  the  police  do  not  have. 

Probably,  when  I  have  gotten  to  the  conclusion  that  our  crews  say  they 
cannot  get  to  it,  I  might  after  the  actual  evacuut ion  of  the  building  lias  been 
completed,  set  up  for  some  type  of  ventilation.  Your  priorities  would  be  just 
on  the  scene,  would  be  your  life.  Concern  yourself  with  the  people  first, 
then  after  that  was  maintained,  or  taken  care  of,  then  I  would  attend  to  the 
actual  fire  problem.  But  that  large  of  a  building,  with  our  standards  as  we 
do  it  now,  would  automatically  be  to  call  for  extra  equipment  and  probably 
medic  crews  also.  I  can’t  remember  if  they  said  they  had  medic  crews 
dispatched  to  this  or  not. 

[INT:  One  medic] 

Okay,  I  would  have  probably  asked  for  additional  medic  crew  being  that  it 
looks  like  there  might  have  been  as  high  as  twenty  people  in  that  building. 
Then  you  just  deal  with  the  problems  as  they  occur.  But  my  initial  size  up  of 
the  sil.ua  Li  on  would  luive*  lit  err  i  as  sue;h. 
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Probe  Point  1 . 1 


Weil,  I  have  to  assume  that  because  Rescue  1  laid  the  line  down  the 
basement,  I  have  to  assume  it  is  like  an  engine,  probably  with  just  additional 
tools  than  what  a  normal  engine  crew  would  carry.  The  heavy  type  tools,  power 
tools  to  effect  a  rescue  and  that  type  thing.  Other  than  that,  I  assume  from 
the  way  Rescue  1  was  used  that  it  is  like  an  engine  only  with  additional 
equipment . 

I  have  fire  coming  out  of  the  basement  out  of  the  middle  of  the  building. 
I  wish  I  knew  this  building  a  little  bit  better.  Wait  a  minute,  they  said 
there  were  two  apartments  in  the  basement. 

Now  I  got  police  doing  rescue  in  the  rest  of  the  building.  I  don’t  see 
any  other  doors  to  the  building  so  I  am  going  to  have  to  assume  there  is  a 
door  at  this  end  and  maybe  one  at  the  other  end.  So  we  have  doors  up  there. 

The  second  crew  on  the  scene,  I  don’t  remember  who  that  was ,  I  am  going 
to  try  to  get  into  the  apartment  or  apartments  in  the  neighborhood  of  the  fire 
to  try  to  stop  extension  that  way.  Third  crew  I  am  going  to  send  down  behind 
rescue  1  to  back  them  up  and  try  to  gain  entry  to  the  fire  area.  It  is  a  long 
haul  down  that  basement.  I  don’t  know  if  both  the  apartments  are  on  one  side 
or  if  there  is  one  apartment  on  each  side  in  the  basement.  Anyhow,  those  two 
crews,  working  together,  should  be  able  to  get  into  the  area  of  the  fire. 

The  forth  crew,  well  really  it  is  going  to  be  my  fifth  crew— 

The  second  crew  really,  go  back  a  little  bit,  first  job  is  to  make  sure 
rescue  1  has  water.  Okay,  then  the  next  available  crew  will  go  to  the  area 
around  over  the  top  of  the  fire,  if  at  all  possible,  try  to  moke  sure  there  is 
no  extension.  The  next  crew,  which  is  really  my  fourth  crew,  send  down  the 
t&sement  to  assist  rescue  1.  Truck  crew,  I  am  going  to  have  trying,  to  look 
and  see  if  there  is  any  other  way  they  can  ventilate  that  basement,  try  to  get 
some  more  smoke  out  of  there.  Also,  I  am  going  to  have  maybe  one  or  two  guys 
out  of  the  truck  crew  start  upstairs  and  check  after  the  police  to  make  sure 
everybody  is  out. 

Then,  let’s  see,  then  I’ve  got  one  medic,  okay.  I  am  going  to — I’ve  got 
everybody  assigned  so  I  am  going  to  stage  maybe  two  more  engines  and  a  truck 
to  have  just  in  case  everything  turns  to  dirt  on  me.  1  want  to  have  somebody 
else  there  that  I  can  assign  quickly  if  I  need  them.  With  that  many  crews  and 
with  fire  limited  the  way  it  is,  I  really  should  not  liave  a  whole  lot  of 
trouble . 
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